Compare commits
116
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5b70dc7357 | ||
|
|
ade3a3c5d4 | ||
|
|
ce06dc8ca5 | ||
|
|
604bbf1a0f | ||
|
|
631d95530a | ||
|
|
d590eda22a | ||
|
|
c75c519e3d | ||
|
|
012006e7a7 | ||
|
|
f983c54e87 | ||
|
|
a4402d4621 | ||
|
|
866392fecd | ||
|
|
287ff3658f | ||
|
|
b250dbb59b | ||
|
|
26089950af | ||
|
|
d1fd7c236f | ||
|
|
18e9856227 | ||
|
|
57fbc1e0b4 | ||
|
|
9cae868a27 | ||
|
|
1e8f79fe18 | ||
|
|
9c6252dace | ||
|
|
8b3ebc78b6 | ||
|
|
2486345679 | ||
|
|
eb35141ecc | ||
|
|
cf0db385f3 | ||
|
|
eafa22a3c3 | ||
|
|
652b5d1e9f | ||
|
|
4fb8f6a711 | ||
|
|
1dbcf2cd9a | ||
|
|
75cf5e3b11 | ||
|
|
8afe783277 | ||
|
|
9cea2bb4f0 | ||
|
|
a0fbe2cdda | ||
|
|
4d2a990de1 | ||
|
|
96a279819e | ||
|
|
443445f0fe | ||
|
|
02e62767a1 | ||
|
|
b4a66b29e2 | ||
|
|
6854eac80e | ||
|
|
6414061022 | ||
|
|
a3aa93534d | ||
|
|
d83f7fadef | ||
|
|
185a90b048 | ||
|
|
78d4f9acbf | ||
|
|
7f67bb80ad | ||
|
|
b21c3eea37 | ||
|
|
f57c95e813 | ||
|
|
a181586139 | ||
|
|
dd2784c0ee | ||
|
|
4473f8c60b | ||
|
|
125e94b429 | ||
|
|
2a943550d4 | ||
|
|
6b45cfd855 | ||
|
|
ed6c084c68 | ||
|
|
91b5105f3c | ||
|
|
7080491f89 | ||
|
|
13ac4b753c | ||
|
|
a5ef464d95 | ||
|
|
dadd6f4769 | ||
|
|
4cc92c1579 | ||
|
|
f40611409d | ||
|
|
e21c35cb7f | ||
|
|
927d7764a4 | ||
|
|
20f36fdc31 | ||
|
|
5089e17ee6 | ||
|
|
07f2dd5a8f | ||
|
|
40329f616d | ||
|
|
fb3a21c332 | ||
|
|
86bc69c0a2 | ||
|
|
6df388b25b | ||
|
|
6ca427096b | ||
|
|
89c435f046 | ||
|
|
af52ea7bcc | ||
|
|
13c0dde538 | ||
|
|
2045b7a0b2 | ||
|
|
5c0027b54b | ||
|
|
87159b0a38 | ||
|
|
227986094b | ||
|
|
9819f0ff12 | ||
|
|
dc44e13bfa | ||
|
|
3ad77095c4 | ||
|
|
0726d9430e | ||
|
|
544a262924 | ||
|
|
187f29e163 | ||
|
|
729468be30 | ||
|
|
de931f784a | ||
|
|
b84c040d27 | ||
|
|
cac435c4b9 | ||
|
|
08997264d2 | ||
|
|
42fd55db08 | ||
|
|
7e532ea982 | ||
|
|
b3198276f7 | ||
|
|
6e1f79e8a9 | ||
|
|
14ffc91dfb | ||
|
|
fbe54ffec5 | ||
|
|
562ae635d7 | ||
|
|
cf3e68e02a | ||
|
|
20e0dd7fe2 | ||
|
|
e99079abf8 | ||
|
|
9b111649f3 | ||
|
|
f207b87ff7 | ||
|
|
c4cbcddeae | ||
|
|
63d608f6fc | ||
|
|
9d00ce2089 | ||
|
|
4502c7a850 | ||
|
|
623348111c | ||
|
|
e084cdd1d6 | ||
|
|
9cd00a877b | ||
|
|
bf54f040f9 | ||
|
|
fe7eb70483 | ||
|
|
c81de3459e | ||
|
|
3f3cf55002 | ||
|
|
190223e030 | ||
|
|
ea6d1d1f00 | ||
|
|
ae65678ba4 | ||
|
|
2e422b66df | ||
|
|
10a0635331 |
@@ -0,0 +1,264 @@
|
||||
---
|
||||
name: issue-to-pr
|
||||
disable-model-invocation: true
|
||||
description: >-
|
||||
End-to-end implement a GitHub issue and open a ready pull request. Invoke
|
||||
explicitly as `/issue-to-pr <number|#ref|url>` (this is a side-effecting
|
||||
workflow — it creates branches and opens PRs — so it does not auto-trigger).
|
||||
It creates an isolated git worktree off the latest main, reproduces the bug
|
||||
(or scopes the feature), finds the root cause, implements the smallest correct
|
||||
change following repo conventions, verifies it (build/lint/tests), runs a Codex
|
||||
adversarial review and addresses the material findings, then commits and opens
|
||||
a ready PR. Not for pure triage (use triage-issue), pure review of an existing
|
||||
PR (use review-pr-or-mr / code-review), or analysis without code changes.
|
||||
---
|
||||
|
||||
# Issue → PR
|
||||
|
||||
Turn a single GitHub issue into a verified, reviewed, ready-to-merge pull
|
||||
request — autonomously — while staying honest about uncertainty and stopping
|
||||
when a human decision is genuinely required.
|
||||
|
||||
The goal is not "produce a diff." It is: **confirm the real problem exists,
|
||||
make the smallest change that correctly solves it, prove it works by execution,
|
||||
have an adversary try to break it, and only then ship a PR.** Speed comes from
|
||||
doing these in order, not from skipping them.
|
||||
|
||||
## Inputs
|
||||
|
||||
- The issue reference is passed as `$ARGUMENTS` — a number (`123`), a `#123`
|
||||
reference, or a full issue URL. If none is given, ask; do not guess.
|
||||
- The repo is the current git repository; PRs target `origin`
|
||||
(`Silo-Server/silo-server`) and base `main`.
|
||||
|
||||
## Operating principles
|
||||
|
||||
- **Default to "maybe nothing needs fixing."** Coding agents "fix" already-correct
|
||||
or stale code in a large fraction of cases because they don't stop to confirm the
|
||||
problem is real. Confirm the issue reproduces on current `main` before changing
|
||||
anything — it may already be fixed.
|
||||
- **Root cause over symptom.** A fix you can't explain is a fix you can't trust.
|
||||
- **Verify by execution, not by reading.** Build/lint/tests are the primary
|
||||
correctness signal. The adversarial review is a *second* layer on top — never a
|
||||
substitute for green checks. If you can't verify it, don't ship it.
|
||||
- **Smallest correct change.** Match surrounding code's idioms, naming, and comment
|
||||
density. Put code in the package that owns the behavior (CLAUDE.md); don't create
|
||||
catch-all helpers or duplicate logic — search for an existing helper before
|
||||
writing a new one, and extract shared logic instead of copying.
|
||||
- **Don't trust recall for APIs.** Verify every symbol, function, and signature you
|
||||
call actually exists by grepping the codebase — agents hallucinate plausible-looking
|
||||
APIs.
|
||||
- **Don't gold-plate.** A reviewer told to find gaps will always find some. Address
|
||||
what affects correctness or the issue's stated requirements; treat the rest as
|
||||
optional. Extra abstraction "just in case" is a defect, not diligence.
|
||||
- **Know when to stop.** See [Stop and ask](#stop-and-ask). Autonomy is for
|
||||
execution, not for inventing product decisions or shipping unverifiable guesses.
|
||||
|
||||
Track the phases below with a task list (TaskCreate/TaskUpdate) so progress is
|
||||
visible and nothing is skipped on long runs.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Understand the issue
|
||||
|
||||
Read everything before writing anything.
|
||||
|
||||
```bash
|
||||
gh issue view <N> --json number,title,body,labels,state,author,url,comments
|
||||
```
|
||||
|
||||
Determine: **bug vs. feature** (labels + body → branch prefix `fix/` vs `feat/`);
|
||||
**acceptance criteria** (what does "done" mean?); **linked scope** (does it serve
|
||||
an epic / sub-issue to reference as `Part of #NNN`?); and **client surface** (does
|
||||
it touch API/auth/playback/session/library/metadata behavior consumed by
|
||||
`silo-android` / `silo-apple`? — if so, coordinated client follow-up may be needed).
|
||||
|
||||
If the issue is vague, underspecified, or needs a product call, stop and ask now —
|
||||
before spending a worktree on the wrong problem.
|
||||
|
||||
### 2. Create a worktree off the latest main
|
||||
|
||||
Work in isolation so the user's checkout is untouched. Create it at the **main
|
||||
repository root**, not inside the current worktree.
|
||||
|
||||
```bash
|
||||
# Resolve the primary repo root (works from inside any worktree)
|
||||
MAIN_ROOT="$(cd "$(dirname "$(git rev-parse --git-common-dir)")" && pwd)"
|
||||
git -C "$MAIN_ROOT" fetch origin # always branch off fresh main
|
||||
|
||||
SLUG=<short-kebab-slug-from-issue-title>
|
||||
BRANCH=fix/issue-<N>-$SLUG # use feat/ for features
|
||||
WT="$MAIN_ROOT/.claude/worktrees/issue-<N>"
|
||||
|
||||
git -C "$MAIN_ROOT" worktree add -b "$BRANCH" "$WT" origin/main
|
||||
cd "$WT"
|
||||
```
|
||||
|
||||
If the branch/worktree already exists from a prior run, reuse it (cd in) rather
|
||||
than failing — but confirm it's based on current `origin/main`; if it's stale, tell
|
||||
the user before continuing. From here, **all work happens in `$WT`** (the Bash tool
|
||||
keeps the working directory between calls); use absolute paths when in doubt.
|
||||
|
||||
### 3. Reproduce / confirm — a hard gate
|
||||
|
||||
This is the highest-leverage step. Do not skip it.
|
||||
|
||||
- **Bug:** reproduce it concretely — ideally a **test that fails on current `main`**
|
||||
(red), or a precise script/trace through the faulty path. Pin down the exact code
|
||||
path and the invariant it violates.
|
||||
- **If you cannot reproduce it, STOP and report** — do not patch. The issue may
|
||||
already be fixed, be environment-specific, or be a misunderstanding. Shipping a
|
||||
speculative fix for a non-reproducing bug is the most common way agents make
|
||||
things worse.
|
||||
- **Feature:** confirm the capability is actually missing, then map where it fits —
|
||||
the owning package, patterns to follow, the data flow, the smallest set of files.
|
||||
|
||||
**Plan proportionally.** If you could describe the diff in one sentence (a typo, a
|
||||
log line, a one-line guard), skip formal planning and make the change. Plan first
|
||||
when the work is multi-file, unfamiliar, or architecturally uncertain — name the
|
||||
files/interfaces involved, state what's out of scope, and end the plan with the
|
||||
verification step. For anything large, confirm direction with the user before a big
|
||||
build-out.
|
||||
|
||||
### 4. Implement
|
||||
|
||||
Make the change, scoped to this one concern (one concern per PR). Respect the hard
|
||||
rules:
|
||||
|
||||
- **API:** additive-only within `/api/v1` — never rename/remove a response field,
|
||||
change a field's type, or repurpose a status code. New behavior = new
|
||||
fields/endpoints; expose capability endpoints for feature detection.
|
||||
- **Migrations:** new DB changes are Goose SQL migrations via
|
||||
`make migrate-create NAME=...` (timestamped). Never `goose fix`, never paired
|
||||
`.up.sql`/`.down.sql`.
|
||||
- **Maintainability:** extract shared logic instead of duplicating; change existing
|
||||
code rather than bolting on local workarounds.
|
||||
|
||||
For a bug, the failing test from step 3 should now pass.
|
||||
|
||||
### 5. Verify — the primary correctness gate
|
||||
|
||||
Run the checks and make them genuinely pass. The exact commands and the worktree
|
||||
quirks (`GOWORK=off`, stubbing `web/dist`, the v2 golangci invocation, the flaky
|
||||
GPU tests) are in **[references/verification.md](references/verification.md)** —
|
||||
read it first, because plain `go build ./...` / `make lint` fail inside
|
||||
`.claude/worktrees`.
|
||||
|
||||
At minimum: Go build + the relevant package tests pass; if `web/` changed,
|
||||
`pnpm run lint` + `pnpm run format:check` + build pass; `make verify-local-paths` is
|
||||
clean. Run the **full relevant suite**, not just your new test — catch regressions
|
||||
in code you didn't touch. Show the command output as evidence; don't assert success.
|
||||
|
||||
**Never reach green by weakening the checks.** Deleting/skipping/`xfail`-ing tests,
|
||||
loosening assertions, lowering coverage thresholds, or relaxing lint/CI to make a
|
||||
red check pass is a blocker, full stop — if a test genuinely must change, that needs
|
||||
explicit human sign-off, not a quiet edit. If you can't make the checks pass on the
|
||||
merits, stop and report.
|
||||
|
||||
### 6. Adversarial review (Codex)
|
||||
|
||||
Commit your work first (so the review sees exactly what will ship), then run a Codex
|
||||
adversarial review of the branch against `main`. Full invocation, output handling,
|
||||
the iteration cap, and the fallback are in
|
||||
**[references/codex-review.md](references/codex-review.md)** — follow it exactly.
|
||||
|
||||
The loop, in short:
|
||||
1. Commit the implementation on the branch.
|
||||
2. Run the adversarial review (`--base main`, foreground/`--wait`, extended timeout).
|
||||
3. For each finding: **first confirm it's real and grounded in the actual code**
|
||||
(reviewers over-flag — don't fix phantom nits). Then **fix** material findings
|
||||
that affect correctness or the stated requirements (and re-run the review), or
|
||||
**rebut** them with a defensible, written reason. Note non-material findings as
|
||||
optional.
|
||||
4. **Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
|
||||
sharply after ~2–3 attempts, so thrashing won't help — if material findings still
|
||||
stand after the cap, stop and escalate to the user instead of looping.
|
||||
|
||||
Do not open the PR while a material, un-rebutted adversarial finding stands.
|
||||
|
||||
### 7. Commit, push, open the PR
|
||||
|
||||
Conventional Commit subjects scoped to the domain, e.g.
|
||||
`fix(playback): guard against nil session on reconnect`. End commit messages with:
|
||||
|
||||
```
|
||||
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||||
```
|
||||
|
||||
Push and open a **ready** (non-draft) PR against `main`:
|
||||
|
||||
```bash
|
||||
git push -u origin "$BRANCH"
|
||||
gh pr create --base main --head "$BRANCH" --title "<conventional subject>" --body "$(cat <<'EOF'
|
||||
## Problem
|
||||
<what's broken / missing, grounded in the issue — and how you reproduced it>
|
||||
|
||||
## Approach
|
||||
<what you changed and *why this approach* — the tradeoff, not a file list>
|
||||
|
||||
## Verification
|
||||
<commands run and their result; the failing→passing test; manual checks>
|
||||
|
||||
## Adversarial review
|
||||
<clean, or the material findings and how each was fixed/rebutted>
|
||||
|
||||
## Risks & follow-up
|
||||
<rollout/migration/compatibility risks; any silo-android / silo-apple
|
||||
follow-up; anything deferred>
|
||||
|
||||
Closes #<N>
|
||||
<add "Part of #<epic>" if the issue serves a capability epic/sub-issue>
|
||||
|
||||
## AI-use disclosure
|
||||
Implemented by Claude Code (issue-to-pr skill) with human oversight.
|
||||
EOF
|
||||
)"
|
||||
```
|
||||
|
||||
End the PR body with:
|
||||
|
||||
```
|
||||
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||||
```
|
||||
|
||||
For **UI changes**, attach before/after screenshots or a recording (the repo
|
||||
requires this); if you can't capture them, explicitly flag in the PR that they're
|
||||
still needed.
|
||||
|
||||
### 8. Report back
|
||||
|
||||
Give the user the PR URL, a 2–3 line summary, the verification result, the
|
||||
adversarial-review outcome, any flagged risks or required client follow-up, and the
|
||||
worktree path so they can inspect it.
|
||||
|
||||
## Long or multi-iteration runs
|
||||
|
||||
Performance degrades on long sessions, and context compaction can silently drop
|
||||
constraints. So: keep a short progress note and a "failed approaches" list in the
|
||||
scratchpad so you don't repeat dead ends; and after any compaction, re-read CLAUDE.md
|
||||
and re-assert the hard guardrails (additive-only API, migration rules, no weakening
|
||||
of tests) before continuing.
|
||||
|
||||
## Stop and ask
|
||||
|
||||
Pause and ask the user (don't push a PR) when:
|
||||
|
||||
- The issue is ambiguous, underspecified, or needs a product/UX decision.
|
||||
- **The bug doesn't reproduce** on current `main`.
|
||||
- The fix would require a non-additive API change, or a risky/irreversible migration.
|
||||
- The change balloons into something large or architectural — propose a plan first.
|
||||
- Checks can't be made to pass on the merits, or material adversarial findings remain
|
||||
after the iteration cap.
|
||||
- A test/CI change would be needed to "pass" — get sign-off, don't do it silently.
|
||||
- The right fix clearly belongs in a sibling repo (`silo-android`, `silo-apple`, a
|
||||
plugin/SDK repo) — see CLAUDE.md's multi-repo guidance.
|
||||
|
||||
In these cases, summarize what you found and what you'd do, and let the user decide.
|
||||
|
||||
## Notes
|
||||
|
||||
- Authorized to open ready PRs autonomously once checks pass and the adversarial
|
||||
review is clean — that's the durable instruction from setup. Not authorized to
|
||||
merge, force-push shared branches, or close issues by hand.
|
||||
- Never invent a fix you can't verify or explain. An honest "here's what's uncertain"
|
||||
beats a confident wrong PR.
|
||||
@@ -0,0 +1,88 @@
|
||||
# Codex adversarial review
|
||||
|
||||
The `/codex:adversarial-review` command is marked `disable-model-invocation: true`,
|
||||
so you cannot trigger it as a tool/skill. Call the same underlying runtime
|
||||
directly via Bash: the `codex-companion.mjs` helper, subcommand
|
||||
`adversarial-review`. It challenges the *approach and design*, not just defects —
|
||||
it tries to find the strongest reasons the change should not ship.
|
||||
|
||||
## 1. Locate the companion script
|
||||
|
||||
```bash
|
||||
CODEX_COMPANION="${CLAUDE_PLUGIN_ROOT:+$CLAUDE_PLUGIN_ROOT/scripts/codex-companion.mjs}"
|
||||
if [ -z "$CODEX_COMPANION" ] || [ ! -f "$CODEX_COMPANION" ]; then
|
||||
CODEX_COMPANION="$(find "$HOME/.claude/plugins/marketplaces" "$HOME/.claude/plugins/cache" \
|
||||
-name codex-companion.mjs 2>/dev/null | head -n1)"
|
||||
fi
|
||||
echo "$CODEX_COMPANION"
|
||||
```
|
||||
|
||||
If nothing is found, the Codex plugin isn't installed/set up. Don't silently skip
|
||||
the review — tell the user, point them at `/codex:setup`, and ask whether to
|
||||
proceed without it (or fall back to a `feature-dev:code-reviewer` agent pass).
|
||||
The user asked specifically for the Codex adversarial review, so treat its absence
|
||||
as a real blocker, not a detail.
|
||||
|
||||
## 2. Commit first, then review the branch against main
|
||||
|
||||
Review what will actually ship. Commit the implementation on the branch, then
|
||||
diff against `main`:
|
||||
|
||||
```bash
|
||||
git add -A && git commit -m "<conventional subject>"
|
||||
```
|
||||
|
||||
Run it in the **foreground** so results come back this turn, and set an extended
|
||||
Bash timeout (reviews can take several minutes) — pass `timeout: 600000` on the
|
||||
Bash call:
|
||||
|
||||
```bash
|
||||
node "$CODEX_COMPANION" adversarial-review --wait --base main "optional focus text"
|
||||
```
|
||||
|
||||
- `--wait` runs foreground and prints the result to stdout.
|
||||
- `--base main` reviews the whole branch diff (`main...HEAD`).
|
||||
- Append focus text to steer it (e.g. the specific failure mode, the risky
|
||||
concurrency path, the migration). Keep the adversarial framing — don't soften it.
|
||||
- Do **not** use `--background` here; you need the findings in-hand to act on them.
|
||||
|
||||
## 3. Interpret the output
|
||||
|
||||
The review returns a verdict plus findings:
|
||||
|
||||
- **`approve` / no material findings** → green light for the PR.
|
||||
- **`needs-attention`** → there is at least one material risk. Each finding names
|
||||
a file, a line range, an impact, and a concrete recommendation.
|
||||
|
||||
LLM reviewers systematically **over-flag** — they report gaps even when the work is
|
||||
sound. So for every finding, in order:
|
||||
|
||||
1. **Confirm it's real.** Open the cited file/lines and verify the finding is
|
||||
grounded in the actual code and actually reachable. Drop findings that reference
|
||||
code that doesn't exist, lines outside the diff, or behavior the code doesn't have
|
||||
— don't fix phantom nits.
|
||||
2. For a confirmed finding, do one of two things — never silently ignore it:
|
||||
- **Fix it** if it affects correctness or the issue's stated requirements, then
|
||||
re-run the review (go back to step 2 after committing the fix); or
|
||||
- **Rebut it** with a defensible, written reason it doesn't apply, and capture
|
||||
that reasoning in the PR's "Adversarial review" section.
|
||||
3. Treat style/naming/speculative/low-value findings as **optional** — don't
|
||||
gold-plate. The adversarial prompt is told not to emit these, but if any slip
|
||||
through, note and move on; don't add abstraction or defensive code for cases that
|
||||
can't happen.
|
||||
|
||||
## 4. Loop — but cap it
|
||||
|
||||
Repeat implement → verify → commit → review until the review is `approve` or every
|
||||
material finding is fixed or defensibly rebutted.
|
||||
|
||||
**Cap the loop at 2 iterations (3 absolute max).** Debugging effectiveness decays
|
||||
sharply after ~2–3 attempts — if material findings still stand after the cap, stop
|
||||
and escalate to the user with a summary rather than thrashing or shipping over a
|
||||
standing objection. Only open the PR once the review is clean within the cap.
|
||||
|
||||
## 5. Record it in the PR
|
||||
|
||||
Summarize the outcome in the PR body's **Adversarial review** section: either
|
||||
"clean — no material findings" or a terse list of what was raised and how each was
|
||||
fixed or rebutted. This gives reviewers the adversary's perspective for free.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Verification inside a worktree
|
||||
|
||||
Commands assume the repository root is the cwd (the new worktree, `$WT`).
|
||||
|
||||
## Why the obvious commands fail here
|
||||
|
||||
A `.claude/worktrees/*` checkout lives under the parent Silo workspace, whose
|
||||
`go.work` only lists the primary `./silo-server`, not worktree checkouts. So:
|
||||
|
||||
- `go env GOWORK` resolves to the parent workspace from anywhere under `SiloServer/`.
|
||||
- Plain `go build ./...` / `go test ./...` fail with
|
||||
*"directory prefix . does not contain modules listed in go.work"*.
|
||||
- `go build ./...` also needs `web/dist` to exist (the embed directive).
|
||||
- `golangci-lint` is not installed locally.
|
||||
|
||||
## Go
|
||||
|
||||
Prefix every Go command with `GOWORK=off`.
|
||||
|
||||
```bash
|
||||
# Build the parts that matter without needing web/dist:
|
||||
GOWORK=off go build ./internal/... ./cmd/...
|
||||
|
||||
# Run the tests for the package(s) you touched (fast, targeted):
|
||||
GOWORK=off go test ./internal/<pkg>/...
|
||||
|
||||
# If you need a full ./... build, stub the embed target first:
|
||||
mkdir -p web/dist && touch web/dist/.gitkeep
|
||||
GOWORK=off go build ./...
|
||||
```
|
||||
|
||||
Lint Go without installing golangci-lint — must be the **v2** module path (the
|
||||
repo's `.golangci` config rejects v1):
|
||||
|
||||
```bash
|
||||
GOWORK=off go run github.com/golangci/golangci-lint/v2/cmd/golangci-lint@latest run ./internal/...
|
||||
```
|
||||
|
||||
**Known flake, not a real failure:** `internal/playback` GPU-probe tests
|
||||
(`TestResolveHWAccel*`, the NVENC smoke test) can fail with `signal: killed`
|
||||
under parallel load on macOS, but pass in isolation. If they're the only
|
||||
failures, re-run them alone before treating them as real:
|
||||
|
||||
```bash
|
||||
GOWORK=off go test ./internal/playback/ -run TestResolveHWAccel -count=1
|
||||
```
|
||||
|
||||
## Frontend (only if `web/` changed)
|
||||
|
||||
```bash
|
||||
cd web
|
||||
pnpm install # if deps changed or node_modules is absent
|
||||
pnpm run lint
|
||||
pnpm run format:check
|
||||
pnpm run build # catches type/build errors before the PR
|
||||
cd ..
|
||||
```
|
||||
|
||||
The repo requires `pnpm run lint` and `pnpm run format:check` to be clean before
|
||||
a merge request.
|
||||
|
||||
## Repo-wide
|
||||
|
||||
```bash
|
||||
make verify-local-paths # must be clean; CLAUDE.md forbids absolute paths in docs/specs
|
||||
```
|
||||
|
||||
## Migrations
|
||||
|
||||
If the change needs schema work, create a timestamped Goose migration — never
|
||||
hand-number, never `goose fix`, never paired up/down files:
|
||||
|
||||
```bash
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
make migrate-status
|
||||
```
|
||||
@@ -76,3 +76,4 @@ POSTGRES_SHM_SIZE=8gb
|
||||
|
||||
# Optional server mode override for source runs or standalone worker nodes.
|
||||
# MODE=integrated
|
||||
#SILO_MIGRATE_TIMEOUT=60m
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
github: quick104
|
||||
+14
-1
@@ -50,7 +50,11 @@ docs/inspiration
|
||||
|
||||
|
||||
.agents/
|
||||
/.claude/
|
||||
# Ignore everything in .claude/ except committed, shared skills.
|
||||
/.claude/*
|
||||
!/.claude/skills/
|
||||
/.claude/skills/*
|
||||
!/.claude/skills/issue-to-pr/
|
||||
.cursor/
|
||||
.superpowers/
|
||||
.codex/
|
||||
@@ -59,5 +63,14 @@ node_modules/
|
||||
logs/
|
||||
.playwright-mcp/
|
||||
demo/
|
||||
docker-compose.dev.yml
|
||||
docker-compose.local.yml
|
||||
# Local-only deployment override (unpublishes bundled redis/postgres host
|
||||
# ports via `ports: !override []`). Kept out of git so a rebase from main
|
||||
# never disturbs it and it never lands in a PR; its absence once exposed
|
||||
# Redis to the internet, so it must persist on the box.
|
||||
docker-compose.override.yml
|
||||
# Local compose customization layered onto docker-compose.yml by make dev-deploy.
|
||||
docker-compose.local.yml
|
||||
.playwright-cli/
|
||||
output/
|
||||
|
||||
@@ -91,10 +91,6 @@ Recent history follows Conventional Commit-style subjects such as `feat(playback
|
||||
|
||||
## Silo v1 Process (scope lock & API rules)
|
||||
|
||||
Silo is working toward a locked v1 feature set so client teams can plan against a stable core. These rules apply to every change.
|
||||
|
||||
**Scope gate.** Before implementing a new user-facing capability, check `docs/architecture/v1-scope.md`. If the capability is not listed there — or the file says scope is not yet locked — do NOT open a feature PR. File a **v1 capability proposal** issue (template in this repo) and wait for triage. Bug fixes, refactors, and work on already-locked capabilities proceed normally.
|
||||
|
||||
**API rules — additive-only within `/api/v1`.**
|
||||
- Never rename or remove a response field, change a field's type, or repurpose a status code on an existing endpoint.
|
||||
- New functionality adds new fields/endpoints; removals go through the existing Deprecation/Sunset header flow only.
|
||||
@@ -103,6 +99,4 @@ Silo is working toward a locked v1 feature set so client teams can plan against
|
||||
|
||||
**PR requirements.**
|
||||
- Link the capability epic or sub-issue the PR serves (`Part of #NNN`). PRs with no linked scope item will be questioned at review.
|
||||
- One concern per PR; Conventional Commit subject; AI-use disclosure in the PR body.
|
||||
|
||||
**Pre-push checklist.** `make lint` · `cd web && pnpm run lint && pnpm run format:check` · `make verify-local-paths` · Go tests in a libvips-capable container (a bare-host `go test ./...` silently skips CGO packages, including `internal/api/handlers`).
|
||||
- One concern per PR; Conventional Commit subject; AI-use disclosure in the PR body.
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
# Developing Silo
|
||||
|
||||
This document covers building, running, and contributing to the Silo server. If you just want to run Silo, see the [README](README.md).
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Go** 1.24+
|
||||
- **Bun** 1.0+
|
||||
- **PostgreSQL** 18+
|
||||
- **FFmpeg** (for transcoding support)
|
||||
|
||||
## Local Development
|
||||
|
||||
Local development remains intentionally separate from the deploy-oriented compose setup. Use [docker-compose.yml](docker-compose.yml) for local services and the source-build workflow below.
|
||||
|
||||
```sh
|
||||
# Start local PostgreSQL and Redis
|
||||
docker compose up -d postgres redis
|
||||
|
||||
# Run the frontend dev server (hot reload, proxies API to :8090)
|
||||
make dev-frontend
|
||||
|
||||
# Run the Go backend
|
||||
make dev-backend
|
||||
```
|
||||
|
||||
The main compose file expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first (`cp .env.example .env`).
|
||||
|
||||
If you are developing `Silo` and `silo-plugin-sdk` together, keep using the local [`go.work`](go.work) workspace. That workspace is a developer convenience only. CI and release builds run with `GOWORK=off`, so any new SDK helper used here must be pushed and tagged in `silo-plugin-sdk` before this repo can merge or release the change.
|
||||
|
||||
Plugin authors should start with [docs/architecture/plugin-development.md](docs/architecture/plugin-development.md), which covers the RPC plugin package format, generated proto workflow, SDK import paths, route and asset exposure, and auth or user-config integration points.
|
||||
|
||||
## Make Targets
|
||||
|
||||
| Target | Description |
|
||||
|---|---|
|
||||
| `make build` | Build frontend + Go binary |
|
||||
| `make frontend` | Build frontend only |
|
||||
| `make dev-frontend` | Vite dev server with HMR |
|
||||
| `make dev-backend` | Run Go backend (integrated mode) |
|
||||
| `make dev-proxy` | Run a standalone proxy node |
|
||||
| `make dev-transcode` | Run a standalone transcode node |
|
||||
| `make migrate-create NAME=add_thing` | Create a timestamped Goose SQL migration |
|
||||
| `make migrate-validate` | Validate Goose migration files without touching a database |
|
||||
| `make migrate-status` | Show Goose migration status using Silo's bootstrapping runner |
|
||||
| `make migrate-up` | Apply pending Goose migrations using Silo's bootstrapping runner |
|
||||
| `make clean` | Remove build artifacts |
|
||||
|
||||
## Database Migrations
|
||||
|
||||
PostgreSQL schema migrations are managed by Goose. Migration SQL files live in
|
||||
`migrations/sql/` and use Goose annotations. Converted legacy migrations keep
|
||||
their original numeric versions so existing `schema_versions` rows can bootstrap
|
||||
cleanly into Goose without replaying old SQL. New migrations should be created
|
||||
with timestamped filenames:
|
||||
|
||||
```sh
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
```
|
||||
|
||||
Do not run `goose fix`; timestamped migrations are the repository policy because
|
||||
they avoid version collisions across parallel PRs. The existing `001`-style
|
||||
files are historical compatibility records, not the naming pattern for new work.
|
||||
Runtime migrations are applied by the integrated/API server only. Proxy and
|
||||
transcode modes never mutate schema.
|
||||
For existing installs, use `make migrate-status` and `make migrate-up` rather
|
||||
than invoking the Goose CLI directly; those targets copy legacy
|
||||
`schema_versions` rows into `public.goose_db_version` under the migration lock
|
||||
before reading or applying migrations. Set `ENV_FILE=path/to/.env` when the
|
||||
database URL should be read from a non-default env file.
|
||||
|
||||
## Running Tests
|
||||
|
||||
```sh
|
||||
# Go tests (uses testcontainers — Docker must be running)
|
||||
go test ./...
|
||||
|
||||
# Frontend tests
|
||||
cd web && bun test
|
||||
```
|
||||
|
||||
## Linting
|
||||
|
||||
```sh
|
||||
# Go
|
||||
golangci-lint run
|
||||
|
||||
# Frontend
|
||||
cd web && bun run lint
|
||||
cd web && bun run format:check
|
||||
```
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
cmd/silo/ Entry point
|
||||
internal/
|
||||
api/ HTTP router, handlers, middleware
|
||||
auth/ JWT authentication and sessions
|
||||
catalog/ Media item, episode, season repositories
|
||||
config/ YAML + env var configuration
|
||||
jellycompat/ Jellyfin/Emby protocol compatibility
|
||||
metadata/ Plugin-driven metadata matching and enrichment
|
||||
playback/ Direct play, remux, transcode session management
|
||||
scanner/ Media file discovery and FFProbe
|
||||
worker/ Background jobs (scan, match, reconcile)
|
||||
web/ React + TypeScript frontend (Vite, Tailwind, shadcn/ui)
|
||||
migrations/sql/ Goose-managed PostgreSQL schema migrations
|
||||
```
|
||||
@@ -1,8 +1,17 @@
|
||||
# Silo
|
||||
|
||||
A self-hosted media streaming server with a React frontend and Go backend. Supports direct play, remuxing, and hardware-accelerated transcoding. Includes optional Jellyfin/Emby-compatible app support for clients such as VidHub and Findroid.
|
||||
Silo is a self-hosted media streaming server for your movies, shows, music, and books. Point it at your media folders and stream to your devices — at home or away — with direct play, remuxing, and hardware-accelerated transcoding handled automatically.
|
||||
|
||||
Join the community on [Discord](https://discord.gg/4RxuUQAEnW).
|
||||
Join the community on [Discord](https://discord.gg/4RxuUQAEnW). If Silo is useful to you, consider [sponsoring the project](https://github.com/sponsors/quick104) — see [Supporting Silo](#supporting-silo).
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Plays your media, your way** — direct play when the device supports it, remux or hardware-accelerated transcode (including NVENC) when it doesn't.
|
||||
- **Web app included** — a full-featured web client and admin interface ship with the server.
|
||||
- **Works with apps you already use** — optional Jellyfin/Emby-compatible API supports clients such as VidHub, Findroid, and Infuse.
|
||||
- **Household profiles** — multiple profiles per account, with per-profile watch state and parental controls.
|
||||
- **Plugin-driven metadata** — match and enrich your libraries with providers like TMDB and TVDB, installed as plugins.
|
||||
- **Fast setup** — one `docker compose up -d` brings up the whole stack; everything else is configured in the admin UI.
|
||||
|
||||
## Deploy with Docker (recommended)
|
||||
|
||||
@@ -105,157 +114,6 @@ Silo is externally stateful by default rather than fully stateless. Durable appl
|
||||
Migrating an existing Continuum Docker install should be done with the preflight
|
||||
helper and cutover guide in [docs/continuum-to-silo-docker-migration.md](docs/continuum-to-silo-docker-migration.md).
|
||||
|
||||
## Build from Source
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- **Go** 1.24+
|
||||
- **Bun** 1.0+
|
||||
- **PostgreSQL** 18+
|
||||
- **FFmpeg** (for transcoding support)
|
||||
|
||||
### Quick Start
|
||||
|
||||
1. **Start PostgreSQL** (skip if you already have one running)
|
||||
|
||||
```sh
|
||||
docker compose up -d postgres redis
|
||||
```
|
||||
|
||||
The main compose file still expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first.
|
||||
|
||||
2. **Configure the database connection**
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and set `DATABASE_URL` to point to your PostgreSQL instance.
|
||||
|
||||
3. **Build and run**
|
||||
|
||||
```sh
|
||||
make build
|
||||
./silo
|
||||
```
|
||||
|
||||
The server starts at `http://localhost:8080` by default. All other settings are configured through the admin UI.
|
||||
|
||||
## Development
|
||||
|
||||
Local development remains intentionally separate from the deploy-oriented compose setup. Use [docker-compose.yml](docker-compose.yml) and the existing source-build workflow for local development.
|
||||
|
||||
```sh
|
||||
# Run the frontend dev server (hot reload, proxies API to :8090)
|
||||
make dev-frontend
|
||||
|
||||
# Run the Go backend
|
||||
make dev-backend
|
||||
```
|
||||
|
||||
If you are developing `Silo` and `silo-plugin-sdk` together, keep using the local [`go.work`](go.work) workspace. That workspace is a developer convenience only. CI and release builds run with `GOWORK=off`, so any new SDK helper used here must be pushed and tagged in `silo-plugin-sdk` before this repo can merge or release the change.
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
Plugin authors should start with [docs/architecture/plugin-development.md](docs/architecture/plugin-development.md), which covers the RPC plugin package format, generated proto workflow, SDK import paths, route and asset exposure, and auth or user-config integration points.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
|
||||
|
||||
Please include:
|
||||
|
||||
- What you were trying to do
|
||||
- Exact steps you took
|
||||
- What you expected to happen
|
||||
- What actually happened
|
||||
- What exact action is slow or broken (`save`, `scan`, `browse`, `import`, `playback`, etc.)
|
||||
- Whether it happens every time or only sometimes
|
||||
- The library, media type, filter, setting, or value involved
|
||||
- Version, branch, commit, and deployment details if you know them
|
||||
- Screenshots, recordings, or log snippets if relevant
|
||||
|
||||
If you used Claude/Codex for debugging, put that under `Technical notes` at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
|
||||
|
||||
Use this template:
|
||||
|
||||
```text
|
||||
Goal:
|
||||
Steps:
|
||||
Expected:
|
||||
Actual:
|
||||
What is slow/broken:
|
||||
Scope:
|
||||
Version/branch:
|
||||
Deployment:
|
||||
Technical notes:
|
||||
```
|
||||
|
||||
### Make Targets
|
||||
|
||||
| Target | Description |
|
||||
|---|---|
|
||||
| `make build` | Build frontend + Go binary |
|
||||
| `make frontend` | Build frontend only |
|
||||
| `make dev-frontend` | Vite dev server with HMR |
|
||||
| `make dev-backend` | Run Go backend (integrated mode) |
|
||||
| `make dev-proxy` | Run a standalone proxy node |
|
||||
| `make dev-transcode` | Run a standalone transcode node |
|
||||
| `make migrate-create NAME=add_thing` | Create a timestamped Goose SQL migration |
|
||||
| `make migrate-validate` | Validate Goose migration files without touching a database |
|
||||
| `make migrate-status` | Show Goose migration status using Silo's bootstrapping runner |
|
||||
| `make migrate-up` | Apply pending Goose migrations using Silo's bootstrapping runner |
|
||||
| `make clean` | Remove build artifacts |
|
||||
|
||||
### Database Migrations
|
||||
|
||||
PostgreSQL schema migrations are managed by Goose. Migration SQL files live in
|
||||
`migrations/sql/` and use Goose annotations. Converted legacy migrations keep
|
||||
their original numeric versions so existing `schema_versions` rows can bootstrap
|
||||
cleanly into Goose without replaying old SQL. New migrations should be created
|
||||
with timestamped filenames:
|
||||
|
||||
```sh
|
||||
make migrate-create NAME=add_thing
|
||||
make migrate-validate
|
||||
```
|
||||
|
||||
Do not run `goose fix`; timestamped migrations are the repository policy because
|
||||
they avoid version collisions across parallel PRs. The existing `001`-style
|
||||
files are historical compatibility records, not the naming pattern for new work.
|
||||
Runtime migrations are applied by the integrated/API server only. Proxy and
|
||||
transcode modes never mutate schema.
|
||||
For existing installs, use `make migrate-status` and `make migrate-up` rather
|
||||
than invoking the Goose CLI directly; those targets copy legacy
|
||||
`schema_versions` rows into `public.goose_db_version` under the migration lock
|
||||
before reading or applying migrations. Set `ENV_FILE=path/to/.env` when the
|
||||
database URL should be read from a non-default env file.
|
||||
|
||||
### Running Tests
|
||||
|
||||
```sh
|
||||
# Go tests (uses testcontainers — Docker must be running)
|
||||
go test ./...
|
||||
|
||||
# Frontend tests
|
||||
cd web && bun test
|
||||
```
|
||||
|
||||
### Linting
|
||||
|
||||
```sh
|
||||
# Go
|
||||
golangci-lint run
|
||||
|
||||
# Frontend
|
||||
cd web && bun run lint
|
||||
cd web && bun run format:check
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Silo is licensed under `AGPL-3.0-or-later`. See [LICENSE](LICENSE).
|
||||
|
||||
## Configuration
|
||||
|
||||
Silo requires only a `DATABASE_URL` when running from source or against external infrastructure. In the default Docker Compose path, the stack wires the database and Redis URLs for you. All other settings — libraries, metadata providers, transcoding, users — are managed through the admin UI after first launch.
|
||||
@@ -269,7 +127,7 @@ Silo requires only a `DATABASE_URL` when running from source or against external
|
||||
| `proxy` | Stream proxy node that connects to the shared deployment database and Redis |
|
||||
| `transcode` | HLS transcode worker node that connects to the shared deployment database and Redis |
|
||||
|
||||
## PostgreSQL Auto-Tuning
|
||||
### PostgreSQL Auto-Tuning
|
||||
|
||||
The default Docker Compose stack does not require a checked-in `postgresql.conf`.
|
||||
It enables Silo's [pgtune](https://github.com/le0pard/pgtune)-style OLTP tuning
|
||||
@@ -321,20 +179,93 @@ change PostgreSQL server settings. Settings already written with `ALTER SYSTEM`
|
||||
remain in `postgresql.auto.conf`; reset those PostgreSQL parameters if you later
|
||||
move fully to a custom `postgresql.conf`.
|
||||
|
||||
## Project Structure
|
||||
## Build from Source
|
||||
|
||||
If you prefer running Silo without Docker:
|
||||
|
||||
1. **Install prerequisites**: Go 1.24+, Bun 1.0+, PostgreSQL 18+, and FFmpeg.
|
||||
|
||||
2. **Start PostgreSQL and Redis** (skip if you already have them running)
|
||||
|
||||
```sh
|
||||
docker compose up -d postgres redis
|
||||
```
|
||||
|
||||
The main compose file still expects `MEDIA_ROOT` to be set even if you only want the bundled PostgreSQL and Redis services, so set that in `.env` first.
|
||||
|
||||
3. **Configure the database connection**
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
```
|
||||
|
||||
Edit `.env` and set `DATABASE_URL` to point to your PostgreSQL instance.
|
||||
|
||||
4. **Build and run**
|
||||
|
||||
```sh
|
||||
make build
|
||||
./silo
|
||||
```
|
||||
|
||||
The server starts at `http://localhost:8080` by default. All other settings are configured through the admin UI.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
If you are reporting a bug, install problem, or performance issue, start with the admin workflow and reproduction steps, not Claude/Codex analysis.
|
||||
|
||||
Please include:
|
||||
|
||||
- What you were trying to do
|
||||
- Exact steps you took
|
||||
- What you expected to happen
|
||||
- What actually happened
|
||||
- What exact action is slow or broken (`save`, `scan`, `browse`, `import`, `playback`, etc.)
|
||||
- Whether it happens every time or only sometimes
|
||||
- The library, media type, filter, setting, or value involved
|
||||
- Version, branch, commit, and deployment details if you know them
|
||||
- Screenshots, recordings, or log snippets if relevant
|
||||
|
||||
If you used Claude/Codex for debugging, put that under `Technical notes` at the end. Suspected files, SQL output, stack traces, and root-cause theories can be helpful, but only after the workflow and repro steps are clear.
|
||||
|
||||
Use this template:
|
||||
|
||||
```text
|
||||
Goal:
|
||||
Steps:
|
||||
Expected:
|
||||
Actual:
|
||||
What is slow/broken:
|
||||
Scope:
|
||||
Version/branch:
|
||||
Deployment:
|
||||
Technical notes:
|
||||
```
|
||||
cmd/silo/ Entry point
|
||||
internal/
|
||||
api/ HTTP router, handlers, middleware
|
||||
auth/ JWT authentication and sessions
|
||||
catalog/ Media item, episode, season repositories
|
||||
config/ YAML + env var configuration
|
||||
jellycompat/ Jellyfin/Emby protocol compatibility
|
||||
metadata/ Plugin-driven metadata matching and enrichment
|
||||
playback/ Direct play, remux, transcode session management
|
||||
scanner/ Media file discovery and FFProbe
|
||||
worker/ Background jobs (scan, match, reconcile)
|
||||
web/ React + TypeScript frontend (Vite, Tailwind, shadcn/ui)
|
||||
migrations/sql/ Goose-managed PostgreSQL schema migrations
|
||||
```
|
||||
|
||||
## Contributing & Development
|
||||
|
||||
Silo is open source and contributions are welcome. See [DEVELOPMENT.md](DEVELOPMENT.md) for building from source in a dev workflow, running tests, database migrations, and project layout, and [CONTRIBUTING.md](CONTRIBUTING.md) for contribution expectations, merge request guidance, and the policy for AI-assisted submissions.
|
||||
|
||||
## Supporting Silo
|
||||
|
||||
Silo is an open-source hobby project, developed in spare time and funded out of pocket. If you'd like to support development, you can sponsor via [GitHub Sponsors](https://github.com/sponsors/quick104).
|
||||
|
||||
Donations go directly toward the costs of building and running the project:
|
||||
|
||||
- AI development tooling subscriptions (Claude, Codex) used to build and maintain Silo
|
||||
- Push notification relay infrastructure
|
||||
- Future development costs
|
||||
|
||||
Sponsoring is entirely optional — Silo is and will remain free and open source. Bug reports, contributions, and feedback are just as valuable.
|
||||
|
||||
## License & Trademarks
|
||||
|
||||
Silo's source code is licensed under the **GNU Affero General Public License
|
||||
v3.0 or later** (`AGPL-3.0-or-later`) — see [LICENSE](LICENSE).
|
||||
|
||||
The **Silo name, logo, and wordmark are trademarks of Silo Media L.L.C.** and
|
||||
are **not** covered by the AGPL. You're free to fork and redistribute the code,
|
||||
but forks and redistributions must not use the Silo brand as their identity and
|
||||
must remove or replace the brand assets. Publishing a Silo-branded app to an app
|
||||
store requires written permission. See [TRADEMARK.md](TRADEMARK.md) for what's
|
||||
permitted — including referential use like "compatible with Silo."
|
||||
|
||||
@@ -0,0 +1,85 @@
|
||||
# Silo Trademark & Brand Policy
|
||||
|
||||
Silo's source code is free and open source under the **GNU Affero General Public
|
||||
License v3.0 or later** (`AGPL-3.0-or-later`). You are welcome to use, study,
|
||||
modify, and redistribute it under those terms.
|
||||
|
||||
**That freedom applies to the code — not to the Silo name and brand.** The Silo
|
||||
trademarks identify the official project and its official builds. This policy
|
||||
explains how they may and may not be used. It exists for a simple reason:
|
||||
anyone should be free to fork the software, while users can still trust that
|
||||
something called "Silo" actually comes from us.
|
||||
|
||||
## Marks covered
|
||||
|
||||
The following are trademarks of **Silo Media L.L.C.** ("Silo Media"), whether or
|
||||
not registered:
|
||||
|
||||
- the names **Silo** and **Silo Server**
|
||||
- the **Silo logo** and **wordmark**
|
||||
- Silo **product and component names**, including **SiloRemote**, **SiloMac**,
|
||||
and other "Silo…" names
|
||||
- confusingly similar variations of the above
|
||||
|
||||
The AGPL license over the source code **does not grant any license to these
|
||||
marks.** This reservation is expressly permitted by AGPL-3.0 §7(e).
|
||||
|
||||
## Uses that are always fine — no permission needed
|
||||
|
||||
- **Referring to Silo truthfully** — "compatible with Silo," "a plugin for
|
||||
Silo," "built on Silo," "a fork of Silo." Use the word to refer to *our*
|
||||
project, not to brand *yours*.
|
||||
- **Redistributing official, unmodified builds** under the Silo name, with all
|
||||
notices kept intact.
|
||||
- **Running, modifying, and self-hosting** Silo for yourself or your
|
||||
organization.
|
||||
- **Editorial and educational** use — articles, tutorials, talks, reviews.
|
||||
|
||||
## Uses that require our written permission
|
||||
|
||||
- Distributing a **modified version under the Silo name** (see *Forks* below).
|
||||
- **Publishing an app to any app store** — Apple App Store, Google Play,
|
||||
F-Droid, or similar — under the Silo name, logo, or a confusingly similar
|
||||
identity.
|
||||
- Using the marks in a **company, product, domain, or app name**, or in any way
|
||||
that implies Silo Media operates or endorses your offering.
|
||||
- **Merchandise** or other commercial use of the logo or wordmark.
|
||||
|
||||
## Forks must rebrand
|
||||
|
||||
You may fork Silo — that is your right under the AGPL. But a **distributed** fork
|
||||
must not use the Silo name or logo as its identity. Pick your own name and logo,
|
||||
remove the Silo brand assets (below), and you may state factually that your
|
||||
project "is based on Silo" or "is a fork of Silo." This is the same model used by
|
||||
Mozilla, Rust, and many other open-source projects: **open code, protected
|
||||
identity.**
|
||||
|
||||
## Brand assets are not under the AGPL
|
||||
|
||||
The Silo logo, wordmark, and icon files are **© 2026 Silo Media L.L.C., all
|
||||
rights reserved**, and are **not** licensed under the AGPL or any other
|
||||
open-source license. Any redistribution or fork must **remove or replace** them.
|
||||
See the `NOTICE` file in the brand-asset directories.
|
||||
|
||||
## Official sources
|
||||
|
||||
The only official Silo software comes from:
|
||||
|
||||
- the **Silo-Server** organization on GitHub — <https://github.com/Silo-Server>
|
||||
- the official website — <https://siloserver.org>
|
||||
- official app-store listings published by **Silo Media L.L.C.**
|
||||
|
||||
Any app-store listing, download, or "Silo" app that does not originate from the
|
||||
above is **not** official and is **not** authorized to use the Silo marks.
|
||||
|
||||
## Requesting permission / reporting misuse
|
||||
|
||||
Email **trademark@siloserver.org** with the details. We're generally glad to
|
||||
grant reasonable requests — the goal of this policy is only to prevent user
|
||||
confusion and unauthorized "official-looking" releases.
|
||||
|
||||
---
|
||||
|
||||
*Silo and the Silo logo are trademarks of Silo Media L.L.C. The **™** symbol is
|
||||
used with the Silo wordmark until federal registration issues, after which **®**
|
||||
applies.*
|
||||
+313
-26
@@ -50,6 +50,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/clientip"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/database"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
"github.com/Silo-Server/silo-server/internal/ebooks"
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
"github.com/Silo-Server/silo-server/internal/historyimport"
|
||||
@@ -57,15 +58,18 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/intromarkers"
|
||||
"github.com/Silo-Server/silo-server/internal/jellycompat"
|
||||
"github.com/Silo-Server/silo-server/internal/libraryingest"
|
||||
"github.com/Silo-Server/silo-server/internal/literaryworks"
|
||||
"github.com/Silo-Server/silo-server/internal/logfilter"
|
||||
"github.com/Silo-Server/silo-server/internal/logstream"
|
||||
"github.com/Silo-Server/silo-server/internal/mail"
|
||||
"github.com/Silo-Server/silo-server/internal/manga"
|
||||
"github.com/Silo-Server/silo-server/internal/markers"
|
||||
"github.com/Silo-Server/silo-server/internal/mdblist"
|
||||
"github.com/Silo-Server/silo-server/internal/metadata"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/nodeconfig"
|
||||
"github.com/Silo-Server/silo-server/internal/nodepool"
|
||||
"github.com/Silo-Server/silo-server/internal/noderecipe"
|
||||
"github.com/Silo-Server/silo-server/internal/nodesessions"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/Silo-Server/silo-server/internal/opslog"
|
||||
@@ -93,6 +97,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/userdb"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore/pgstore"
|
||||
"github.com/Silo-Server/silo-server/internal/watchlist"
|
||||
"github.com/Silo-Server/silo-server/internal/watchstate"
|
||||
"github.com/Silo-Server/silo-server/internal/watchsync"
|
||||
watchmdblist "github.com/Silo-Server/silo-server/internal/watchsync/providers/mdblist"
|
||||
@@ -518,6 +523,12 @@ func main() {
|
||||
// still encrypts/decrypts — no raw settings repo may escape into later wiring.
|
||||
settingsRepo = catalog.NewEncryptedSettingsRepo(catalog.NewServerSettingsRepo(pool), dataCipher)
|
||||
nodeID := resolveNodeIdentity()
|
||||
catalogSearchStartupSettings, err := catalog.CatalogSearchSettingsFromMap(settings)
|
||||
if err != nil {
|
||||
slog.Warn("catalog search: failed to load settings for startup wiring; using postgres", "err", err)
|
||||
catalogSearchStartupSettings = catalog.DefaultCatalogSearchSettings()
|
||||
}
|
||||
activeCatalogSearchProvider := catalog.ActiveCatalogSearchProvider(catalogSearchStartupSettings)
|
||||
|
||||
// Step 9: Validate
|
||||
if err := cfg.Validate(); err != nil {
|
||||
@@ -606,6 +617,10 @@ func main() {
|
||||
} else {
|
||||
srv := transcodenode.NewServer(watcher, tracker)
|
||||
srv.SetFFmpegLogSink(playback.NewSlogFFmpegLogSink(slog.Default(), nodeID))
|
||||
// Read jellycompat reconstruction recipes central wrote at transcode
|
||||
// start, so this node can rebuild a Jellyfin transcode after its own
|
||||
// restart (the node hop token is recipe-less). Shares the offload Redis.
|
||||
srv.SetRecipeStore(noderecipe.NewStore(redisClient, 0))
|
||||
handler = srv.Handler()
|
||||
}
|
||||
|
||||
@@ -802,6 +817,11 @@ func main() {
|
||||
configureS3Clients(cfg, &deps)
|
||||
}
|
||||
|
||||
var literaryWorkService *literaryworks.Service
|
||||
if deps.DB != nil {
|
||||
literaryWorkService = literaryworks.NewService(literaryworks.NewRepository(deps.DB))
|
||||
}
|
||||
|
||||
// Step 4: Create scanner (if needed).
|
||||
if needsScanner && deps.DB != nil {
|
||||
folderRepo := catalog.NewFolderRepository(deps.DB)
|
||||
@@ -811,9 +831,11 @@ func main() {
|
||||
|
||||
ffprobePath := scanner.FFprobePathFromFFmpeg(cfg.Playback.FFmpegPath)
|
||||
s := scanner.NewScanner(fileRepo, ffprobePath, deps.S3Public, cfg.Scanner.Workers, cfg.Scanner.EmptyTrashAfterScan)
|
||||
s.SetSearchIndexProvider(activeCatalogSearchProvider)
|
||||
configWatcher.OnChange(func(_, updated *config.Config) {
|
||||
s.SetWorkers(updated.Scanner.Workers)
|
||||
})
|
||||
s.SetLiteraryWorkLinker(literaryWorkService)
|
||||
deps.Scanner = s
|
||||
deps.ProbeEnsurer = scanner.NewPlaybackProbeEnsurer(fileRepo, ffprobePath, 10*time.Second)
|
||||
slog.Info("scanner initialized")
|
||||
@@ -1034,6 +1056,7 @@ func main() {
|
||||
|
||||
// Step 4b: Create metadata service and match worker (if needed).
|
||||
var metadataService *metadata.MetadataService
|
||||
var metadataImageCacheProcessor *metadata.ImageCacheProcessor
|
||||
var personRefreshService *metadata.PersonRefreshService
|
||||
var matchWorker *metadata.MatchWorker
|
||||
var libraryIngestExecutor *libraryingest.Executor
|
||||
@@ -1051,10 +1074,11 @@ func main() {
|
||||
var episodeRepo *catalog.EpisodeRepository
|
||||
var audiobookEnricher *audiobooks.Enricher
|
||||
var ebookEnricher *ebooks.Enricher
|
||||
var mangaEnricher *manga.Enricher
|
||||
if needsWorkers && deps.DB != nil && deps.FileRepo != nil {
|
||||
chainRepo := metadata.NewChainRepository(deps.DB)
|
||||
skippedRootRepo = metadata.NewSkippedRootRepository(deps.DB)
|
||||
itemRepo = catalog.NewItemRepository(deps.DB)
|
||||
itemRepo = catalog.NewItemRepository(deps.DB).WithActiveSearchProvider(activeCatalogSearchProvider)
|
||||
episodeRepo = catalog.NewEpisodeRepository(deps.DB)
|
||||
seasonRepo = catalog.NewSeasonRepository(deps.DB)
|
||||
personRepo := catalog.NewPersonRepository(deps.DB)
|
||||
@@ -1064,33 +1088,14 @@ func main() {
|
||||
<-pluginAutoUpdateDone
|
||||
|
||||
imageResolver := metadata.NewPluginImageResolver()
|
||||
if pluginService != nil {
|
||||
// Register image resolver sources for all enabled plugin metadata providers.
|
||||
rows, err := deps.DB.Query(appCtx,
|
||||
`SELECT pc.plugin_installation_id, pc.capability_id
|
||||
FROM plugin_capabilities pc
|
||||
JOIN plugin_installations pi ON pi.id = pc.plugin_installation_id
|
||||
WHERE pc.capability_type = 'metadata_provider.v1' AND pi.enabled = true`)
|
||||
if err != nil {
|
||||
slog.Warn("failed to list capabilities for image resolver registration", "error", err)
|
||||
} else {
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var instID int
|
||||
var capID string
|
||||
if err := rows.Scan(&instID, &capID); err != nil {
|
||||
slog.Warn("failed to scan capability for image resolver", "error", err)
|
||||
continue
|
||||
}
|
||||
source := metadata.NewPluginClientSource(instID, capID, func(
|
||||
ctx context.Context, installationID int, capabilityID string,
|
||||
) (metadata.PluginMetadataClient, error) {
|
||||
return pluginService.MetadataProviderClient(ctx, installationID, capabilityID)
|
||||
})
|
||||
imageResolver.RegisterSource(capID, source)
|
||||
slog.Info("registered plugin image resolver", "capability_id", capID, "installation_id", instID)
|
||||
if pluginService != nil && pluginInstallationStore != nil {
|
||||
reloadImageResolvers := func(ctx context.Context) {
|
||||
if err := reloadPluginImageResolvers(ctx, pluginInstallationStore, imageResolver, pluginService); err != nil {
|
||||
slog.Warn("failed to reload plugin image resolvers", "error", err)
|
||||
}
|
||||
}
|
||||
pluginService.AddLifecycleHook(reloadImageResolvers)
|
||||
reloadImageResolvers(appCtx)
|
||||
}
|
||||
if deps.S3Public != nil {
|
||||
presignTTL := cfg.S3.MetadataPresignExpiry
|
||||
@@ -1140,6 +1145,16 @@ func main() {
|
||||
personRepo,
|
||||
providerIDRepo,
|
||||
)
|
||||
audiobookEnricher.SetLiteraryWorkLinker(literaryWorkService)
|
||||
ebookEnricher.SetLiteraryWorkLinker(literaryWorkService)
|
||||
mangaEnricher = manga.NewEnricher(
|
||||
deps.DB,
|
||||
chainRepo,
|
||||
pluginResolver,
|
||||
itemRepo,
|
||||
personRepo,
|
||||
providerIDRepo,
|
||||
)
|
||||
|
||||
// Always wire the image resolver so plugin-prefixed URLs (e.g.
|
||||
// metadb://) can be resolved to presigned HTTP URLs in API responses.
|
||||
@@ -1150,23 +1165,47 @@ func main() {
|
||||
if deps.S3Public != nil {
|
||||
imageCacher := imagecache.New(deps.S3Public)
|
||||
metadataService.SetImageCacher(imageCacher)
|
||||
imageCacheJobs := metadata.NewImageCacheJobRepository(deps.DB)
|
||||
metadataService.SetImageCacheJobEnqueuer(imageCacheJobs)
|
||||
metadataImageCacheProcessor = metadata.NewImageCacheProcessorWithTargets(
|
||||
imageCacheJobs,
|
||||
imageCacher,
|
||||
imageResolver,
|
||||
metadata.ImageCacheProcessorTargets{
|
||||
Items: itemRepo,
|
||||
Seasons: seasonRepo,
|
||||
Episodes: episodeRepo,
|
||||
ItemLocalizations: catalog.NewMediaItemLocalizationRepository(deps.DB),
|
||||
SeasonLocalizations: catalog.NewSeasonLocalizationRepository(deps.DB),
|
||||
People: personRepo,
|
||||
},
|
||||
)
|
||||
metadataService.SetAutoCacheImages(cfg.Metadata.CacheImages)
|
||||
metadataImageCacheProcessor.SetEnabled(cfg.Metadata.CacheImages)
|
||||
configWatcher.OnChange(func(_, updated *config.Config) {
|
||||
metadataService.SetAutoCacheImages(updated.Metadata.CacheImages)
|
||||
metadataImageCacheProcessor.SetEnabled(updated.Metadata.CacheImages)
|
||||
})
|
||||
if deps.Scanner != nil {
|
||||
deps.Scanner.SetImageCacher(imageCacher)
|
||||
}
|
||||
if cfg.Metadata.CacheImages {
|
||||
personRefreshService.SetImageCacher(imageCacher)
|
||||
personRefreshService.SetImageCacheJobEnqueuer(imageCacheJobs)
|
||||
slog.Info("metadata image caching enabled")
|
||||
}
|
||||
if audiobookEnricher != nil {
|
||||
audiobookEnricher.SetImageCacher(imageCacher)
|
||||
audiobookEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
|
||||
audiobookEnricher.SetFFmpegPath(scanner.FFmpegPathFromFFprobe(scanner.FFprobePathFromFFmpeg(cfg.Playback.FFmpegPath)))
|
||||
}
|
||||
if ebookEnricher != nil {
|
||||
ebookEnricher.SetImageCacher(imageCacher)
|
||||
ebookEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
|
||||
}
|
||||
if mangaEnricher != nil {
|
||||
mangaEnricher.SetImageCacher(imageCacher)
|
||||
mangaEnricher.SetImageCacheJobEnqueuer(imageCacheJobs)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1365,6 +1404,7 @@ func main() {
|
||||
)
|
||||
userStoreProvider = notifications.WrapUserStoreProvider(userStoreProvider, notificationSystem)
|
||||
deps.Notifications = notificationSystem
|
||||
|
||||
if libraryIngestExecutor != nil {
|
||||
libraryIngestExecutor.SetAvailabilityDetector(notificationSystem.Detector)
|
||||
}
|
||||
@@ -1417,6 +1457,15 @@ func main() {
|
||||
}
|
||||
})
|
||||
}
|
||||
// Auto-remove fully-watched items from the watchlist (standalone behavior,
|
||||
// default-on per profile), propagating removals to connected providers.
|
||||
if itemRepo != nil && episodeRepo != nil && userStoreProvider != nil {
|
||||
maintainer := watchlist.NewMaintainer(userStoreProvider, itemRepo, episodeRepo)
|
||||
if watchProviderService != nil {
|
||||
maintainer.WithListEventDispatcher(watchProviderService)
|
||||
}
|
||||
deps.WatchCompletionObserver = maintainer
|
||||
}
|
||||
deps.SessionMgr = sessionMgr
|
||||
deps.PlaybackRealtimeHub = playback.NewRealtimeHub()
|
||||
if chapterThumbService != nil && deps.S3Public != nil {
|
||||
@@ -1480,6 +1529,7 @@ func main() {
|
||||
cfg.Recommendations,
|
||||
)
|
||||
deps.Recommender = recEngine
|
||||
deps.CatalogSearchVectorizer = recEngine
|
||||
|
||||
var err error
|
||||
recWorker, err = recommendations.NewWorker(
|
||||
@@ -1675,6 +1725,9 @@ func main() {
|
||||
taskMgr.Register(tasks.NewScanLibrariesTask(deps.FolderRepo, deps.LibraryScanQueue, deps.EventBus))
|
||||
}
|
||||
taskMgr.Register(tasks.NewCleanupOrphanedMediaItemsTask(catalog.NewOrphanedProvisionalCleaner(deps.DB)))
|
||||
catalogSearchIndexer := catalog.NewCatalogSearchIndexer(deps.DB, settingsRepo)
|
||||
taskMgr.Register(tasks.NewSyncCatalogSearchIndexTask(catalogSearchIndexer))
|
||||
taskMgr.Register(tasks.NewRebuildCatalogSearchIndexTask(catalogSearchIndexer))
|
||||
if deps.IntroAnalyzer != nil {
|
||||
taskMgr.Register(tasks.NewDetectIntroMarkersTask(deps.IntroAnalyzer, settingsRepo))
|
||||
}
|
||||
@@ -1688,6 +1741,41 @@ func main() {
|
||||
}
|
||||
taskMgr.Register(tasks.NewActivityLogCleanupTask(deps.DB, settingsRepo, activityPM))
|
||||
taskMgr.Register(tasks.NewOperationalLogCleanupTask(deps.DB, settingsRepo, opsPM))
|
||||
if deps.FileRepo != nil {
|
||||
// Download prepare-to-file pipeline (Phase 3): a durable, leased encode
|
||||
// queue hosted on the task manager. Built here (before Start) and shared
|
||||
// with the API via deps so the download service can enqueue jobs.
|
||||
artifactMgr := downloads.NewArtifactManager(
|
||||
downloads.NewArtifactRepository(deps.DB),
|
||||
downloads.NewRepository(deps.DB),
|
||||
deps.FileRepo,
|
||||
downloads.NewPlaybackPreparer(),
|
||||
deps.NodeID,
|
||||
func() *config.Config {
|
||||
if deps.LiveConfig != nil {
|
||||
if c := deps.LiveConfig(); c != nil {
|
||||
return c
|
||||
}
|
||||
}
|
||||
return deps.Config
|
||||
},
|
||||
func(ctx context.Context, d *downloads.Download) {
|
||||
if deps.EventsHub == nil {
|
||||
return
|
||||
}
|
||||
_ = deps.EventsHub.PublishJSON(ctx, evt.ChannelUserState, "download", map[string]any{
|
||||
"download_id": d.ID,
|
||||
"status": d.Status,
|
||||
"media_item_id": d.ContentID,
|
||||
"format": d.Format,
|
||||
}, evt.PublishOptions{UserID: d.UserID, ProfileID: d.ProfileID})
|
||||
},
|
||||
)
|
||||
encodeTask := tasks.NewEncodeDownloadArtifactsTask(artifactMgr)
|
||||
artifactMgr.SetKick(func() { _ = taskMgr.RunTask(appCtx, encodeTask.Key()) })
|
||||
taskMgr.Register(encodeTask)
|
||||
deps.ArtifactManager = artifactMgr
|
||||
}
|
||||
if notificationSystem != nil {
|
||||
taskMgr.Register(tasks.NewSeedContentAvailabilityTask(notificationSystem))
|
||||
taskMgr.Register(tasks.NewRebuildReleaseInterestTask(notificationSystem))
|
||||
@@ -1699,6 +1787,9 @@ func main() {
|
||||
if refreshWorker != nil && metadataService != nil {
|
||||
taskMgr.Register(tasks.NewRefreshMetadataTask(refreshWorker, metadataService))
|
||||
}
|
||||
if metadataImageCacheProcessor != nil {
|
||||
taskMgr.Register(tasks.NewCacheMetadataImagesTask(metadataImageCacheProcessor))
|
||||
}
|
||||
if pluginAutoUpdater != nil {
|
||||
taskMgr.Register(tasks.NewCheckPluginUpdatesTask(pluginAutoUpdater))
|
||||
}
|
||||
@@ -1775,6 +1866,9 @@ func main() {
|
||||
if ebookEnricher != nil {
|
||||
taskMgr.Register(tasks.NewSyncEbookMetadataTask(ebookEnricher))
|
||||
}
|
||||
if mangaEnricher != nil {
|
||||
taskMgr.Register(tasks.NewSyncMangaMetadataTask(mangaEnricher))
|
||||
}
|
||||
if pluginInstallationStore != nil && pluginRuntimeConfigStore != nil && pluginService != nil {
|
||||
pluginTasks, err := plugins.NewTaskRegistryWithTypedResolver(pluginInstallationStore, pluginRuntimeConfigStore, pluginService).Tasks(appCtx)
|
||||
if err != nil {
|
||||
@@ -1839,6 +1933,8 @@ func main() {
|
||||
AccessResolver: audiobooks.NewABSAccessResolver(absUserRepo, userStoreProvider),
|
||||
Recs: recommendations.NewRepo(deps.DB),
|
||||
Detail: absDetailSvc,
|
||||
SessionMgr: sessionMgr,
|
||||
SessionSyncer: deps.SessionSyncer,
|
||||
}
|
||||
absH := audiobooksService.BuildABSHandler(absHDeps)
|
||||
deps.ABSHandler = absH
|
||||
@@ -2092,6 +2188,11 @@ func main() {
|
||||
NodePlanner: deps.NodePlanner,
|
||||
JWTSecret: cfg.Auth.JWTSecret,
|
||||
RecWorker: recWorker,
|
||||
FrontendFS: deps.FrontendFS,
|
||||
// Hand remote-transcode recipes to the shared recipe store so a dedicated
|
||||
// transcode node that restarts can rebuild a jellycompat session.
|
||||
RecipeNodeStore: noderecipe.NewStore(apiRedisClient, 0),
|
||||
SessionSyncer: deps.SessionSyncer,
|
||||
}
|
||||
|
||||
// Wire direct dependencies when DB is available.
|
||||
@@ -2127,8 +2228,20 @@ func main() {
|
||||
compatDeps.FolderRepo = folderRepo
|
||||
compatDeps.SessionMgr = sessionMgr
|
||||
compatDeps.UserStoreProvider = userStoreProvider
|
||||
compatDeps.WatchCompletionObserver = deps.WatchCompletionObserver
|
||||
compatDeps.SettingsRepo = settingsRepo
|
||||
compatDeps.PersonRepo = personRepo
|
||||
compatSearchService := catalog.NewCatalogSearchService(
|
||||
appCtx,
|
||||
settingsRepo,
|
||||
itemRepo,
|
||||
catalog.NewSearchIndexEventRepository(deps.DB),
|
||||
deps.CatalogSearchVectorizer,
|
||||
)
|
||||
if compatSearchService != nil {
|
||||
compatSearchService.StartCoverageRefresh(appCtx)
|
||||
compatDeps.CatalogSearchProvider = compatSearchService.Provider()
|
||||
}
|
||||
|
||||
if deps.S3Public != nil {
|
||||
compatDeps.PosterPresigner = deps.S3Public
|
||||
@@ -2433,6 +2546,180 @@ func configureS3Clients(cfg *config.Config, deps *api.Dependencies) {
|
||||
}
|
||||
}
|
||||
|
||||
type pluginImageResolverCapabilityStore interface {
|
||||
ListEnabled(ctx context.Context) ([]*plugins.Installation, error)
|
||||
ListCapabilities(ctx context.Context, installationID int) ([]*plugins.Capability, error)
|
||||
}
|
||||
|
||||
func reloadPluginImageResolvers(
|
||||
ctx context.Context,
|
||||
store pluginImageResolverCapabilityStore,
|
||||
resolver *metadata.PluginImageResolver,
|
||||
service *plugins.Service,
|
||||
) error {
|
||||
if resolver == nil {
|
||||
return nil
|
||||
}
|
||||
if store == nil || service == nil {
|
||||
resolver.ReplaceSources(nil)
|
||||
return nil
|
||||
}
|
||||
|
||||
installations, err := store.ListEnabled(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list enabled plugin installations: %w", err)
|
||||
}
|
||||
sort.Slice(installations, func(i, j int) bool {
|
||||
if installations[i] == nil {
|
||||
return false
|
||||
}
|
||||
if installations[j] == nil {
|
||||
return true
|
||||
}
|
||||
return installations[i].ID < installations[j].ID
|
||||
})
|
||||
|
||||
var registrations []metadata.PluginImageResolverSourceRegistration
|
||||
for _, installation := range installations {
|
||||
if installation == nil {
|
||||
continue
|
||||
}
|
||||
capabilities, err := store.ListCapabilities(ctx, installation.ID)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list image resolver capabilities for installation %d: %w", installation.ID, err)
|
||||
}
|
||||
sort.Slice(capabilities, func(i, j int) bool {
|
||||
if capabilities[i] == nil {
|
||||
return false
|
||||
}
|
||||
if capabilities[j] == nil {
|
||||
return true
|
||||
}
|
||||
if capabilities[i].Type != capabilities[j].Type {
|
||||
return capabilities[i].Type < capabilities[j].Type
|
||||
}
|
||||
return capabilities[i].ID < capabilities[j].ID
|
||||
})
|
||||
|
||||
for _, capability := range capabilities {
|
||||
if capability == nil {
|
||||
continue
|
||||
}
|
||||
switch capability.Type {
|
||||
case sdkcapability.ImageResolver:
|
||||
schemes, priority := imageResolverCapabilityConfig(capability)
|
||||
if len(schemes) == 0 {
|
||||
slog.Warn("plugin image resolver capability has no valid schemes",
|
||||
"installation_id", installation.ID,
|
||||
"capability_id", capability.ID)
|
||||
continue
|
||||
}
|
||||
for _, scheme := range schemes {
|
||||
source := metadata.NewPluginClientSource(installation.ID, capability.ID, func(
|
||||
ctx context.Context, installationID int, capabilityID string,
|
||||
) (metadata.PluginMetadataClient, error) {
|
||||
return service.ImageResolverClient(ctx, installationID, capabilityID)
|
||||
})
|
||||
registrations = append(registrations, metadata.PluginImageResolverSourceRegistration{
|
||||
Scheme: scheme,
|
||||
Source: source,
|
||||
Kind: metadata.PluginImageResolverSourceExplicit,
|
||||
Priority: priority,
|
||||
InstallationID: installation.ID,
|
||||
CapabilityID: capability.ID,
|
||||
})
|
||||
}
|
||||
case sdkcapability.MetadataProvider:
|
||||
scheme := strings.TrimSpace(capability.ID)
|
||||
if !metadata.ValidImageResolverScheme(scheme) {
|
||||
slog.Warn("skipping legacy metadata image resolver with invalid scheme",
|
||||
"installation_id", installation.ID,
|
||||
"capability_id", capability.ID)
|
||||
continue
|
||||
}
|
||||
source := metadata.NewPluginClientSource(installation.ID, capability.ID, func(
|
||||
ctx context.Context, installationID int, capabilityID string,
|
||||
) (metadata.PluginMetadataClient, error) {
|
||||
return service.MetadataProviderClient(ctx, installationID, capabilityID)
|
||||
})
|
||||
registrations = append(registrations, metadata.PluginImageResolverSourceRegistration{
|
||||
Scheme: scheme,
|
||||
Source: source,
|
||||
Kind: metadata.PluginImageResolverSourceLegacy,
|
||||
InstallationID: installation.ID,
|
||||
CapabilityID: capability.ID,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resolver.ReplaceSources(registrations)
|
||||
slog.Info("reloaded plugin image resolvers", "sources", len(registrations))
|
||||
return nil
|
||||
}
|
||||
|
||||
func imageResolverCapabilityConfig(capability *plugins.Capability) ([]string, int) {
|
||||
if capability == nil {
|
||||
return nil, 0
|
||||
}
|
||||
meta := capabilityMetadataFields(capability.Metadata)
|
||||
return metadataStringList(meta["schemes"]), metadataInt(meta["priority"])
|
||||
}
|
||||
|
||||
func capabilityMetadataFields(raw map[string]any) map[string]any {
|
||||
if raw == nil {
|
||||
return nil
|
||||
}
|
||||
if nested, ok := raw["metadata"]; ok {
|
||||
switch typed := nested.(type) {
|
||||
case map[string]any:
|
||||
return typed
|
||||
}
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
func metadataStringList(value any) []string {
|
||||
var out []string
|
||||
switch typed := value.(type) {
|
||||
case []string:
|
||||
for _, item := range typed {
|
||||
if scheme := strings.TrimSpace(item); metadata.ValidImageResolverScheme(scheme) {
|
||||
out = append(out, scheme)
|
||||
}
|
||||
}
|
||||
case []any:
|
||||
for _, item := range typed {
|
||||
text, ok := item.(string)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
if scheme := strings.TrimSpace(text); metadata.ValidImageResolverScheme(scheme) {
|
||||
out = append(out, scheme)
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func metadataInt(value any) int {
|
||||
switch typed := value.(type) {
|
||||
case int:
|
||||
return typed
|
||||
case int32:
|
||||
return int(typed)
|
||||
case int64:
|
||||
return int(typed)
|
||||
case float64:
|
||||
return int(typed)
|
||||
case json.Number:
|
||||
n, _ := typed.Int64()
|
||||
return int(n)
|
||||
default:
|
||||
return 0
|
||||
}
|
||||
}
|
||||
|
||||
type markerPluginCapabilityStore interface {
|
||||
ListEnabled(ctx context.Context) ([]*plugins.Installation, error)
|
||||
ListCapabilities(ctx context.Context, installationID int) ([]*plugins.Capability, error)
|
||||
|
||||
@@ -23,6 +23,9 @@ func buildLiveSessionSync(s *playback.Session, reportingNode string) worker.Sess
|
||||
PlayMethod: string(s.PlayMethod),
|
||||
ReportingNode: reportingNode,
|
||||
ClientIP: s.ClientIP,
|
||||
ClientName: s.ClientName,
|
||||
ClientVersion: s.ClientVersion,
|
||||
ClientUserAgent: s.ClientUserAgent,
|
||||
AudioTrackIndex: s.AudioTrackIndex,
|
||||
TranscodeAudio: s.TranscodeAudio,
|
||||
StreamBitrateKbps: s.StreamBitrateKbps,
|
||||
|
||||
@@ -31,6 +31,26 @@ services:
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
|
||||
meilisearch:
|
||||
image: ${MEILISEARCH_IMAGE:-getmeili/meilisearch:latest}
|
||||
restart: unless-stopped
|
||||
profiles: [search]
|
||||
entrypoint:
|
||||
- /bin/sh
|
||||
- -euc
|
||||
- |
|
||||
if [ -z "$${MEILI_MASTER_KEY:-}" ]; then
|
||||
echo "MEILI_MASTER_KEY is required when enabling the search profile" >&2
|
||||
exit 1
|
||||
fi
|
||||
exec /bin/meilisearch
|
||||
environment:
|
||||
MEILI_MASTER_KEY: ${MEILI_MASTER_KEY:-}
|
||||
ports:
|
||||
- "${MEILISEARCH_HOST:-127.0.0.1}:${MEILISEARCH_PORT:-7700}:7700"
|
||||
volumes:
|
||||
- ${SILO_DATA_ROOT:-/opt/silo}/meilisearch:/meili_data
|
||||
|
||||
silo:
|
||||
image: ${SILO_IMAGE:-ghcr.io/silo-server/silo-server:latest}
|
||||
restart: unless-stopped
|
||||
|
||||
@@ -0,0 +1,787 @@
|
||||
# Restart-resilient playback (TR-lease)
|
||||
|
||||
How Silo keeps playback alive across a server restart, reconnect, or partial
|
||||
stream — and how the design arrived at **TR-lease** (token-carried
|
||||
reconstruction + a server-side session-deny marker for revocation).
|
||||
|
||||
> **Status: the revocation half is deferred to a future PR.** Only the
|
||||
> token-carried **reconstruction** core shipped in this PR. The **session-deny /
|
||||
> stream-revocation** mechanism described throughout (the `silo:streamauth:<sid>`
|
||||
> deny marker, the proxy's `Allowed()` enforcement, and the admin
|
||||
> Stop/Terminate deny write) is **not present in the current implementation**.
|
||||
> Admin Terminate and user Stop tear down the live in-memory session and the
|
||||
> ffmpeg producer, but they do **not** prevent a still-valid stream token from
|
||||
> reconstructing the session until its 24h TTL expires. Sections describing that
|
||||
> mechanism are kept for design continuity and are individually flagged as
|
||||
> deferred below.
|
||||
|
||||
This document is the consolidated design record. It folds together four earlier
|
||||
working notes so a future engineer can see the full evolution, the options
|
||||
weighed, the issues raised across review rounds, the variables traded off, and
|
||||
why the final shape was chosen. The superseded notes were:
|
||||
|
||||
- *unified-playback-reconstruct* — the reconstruct core (PR #174), "RC".
|
||||
- *token-carried-playback-reconstruct* — the storage evolution, "TR".
|
||||
- *async-recipe-card-playback* — a monitoring-preserving middle path, "RC-async".
|
||||
- *playback-reconstruct-options-comparison* — the goals/options matrix.
|
||||
|
||||
Paths are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem
|
||||
|
||||
Before this work, a missing in-memory playback session was a `404`. A server
|
||||
restart (deploy, crash, OOM) wiped the in-memory session map, so every in-flight
|
||||
stream died: the client's next manifest/segment/byte-range request hit a session
|
||||
that no longer existed and playback stopped hard. The same gap appeared on any
|
||||
front-end that had never held the session (horizontal scale) and on a dedicated
|
||||
transcode node reboot.
|
||||
|
||||
Core priorities for the fix (from the repo guidelines): **performance first,
|
||||
reliability first, predictable behavior under load and during failures**. The
|
||||
hot path (per-segment serve) must stay off central/Postgres; bytes must stay
|
||||
offloaded to nodes; correctness and robustness win over convenience.
|
||||
|
||||
---
|
||||
|
||||
## 2. The core insight: reconstruct, don't rehydrate
|
||||
|
||||
A missing session is a **reconstruct trigger, not a 404**. The client always
|
||||
re-requests the next chunk after an outage, and its request already states where
|
||||
it is (an HTTP `Range`, a `?seek=`, or a `seg_NNNNN` number). So the server
|
||||
rebuilds the session just-in-time from two things:
|
||||
|
||||
1. a small **durable descriptor** (identity, ownership, and the byte-affecting
|
||||
encode parameters — the "recipe"), and
|
||||
2. the **position the client supplies** on the triggering request.
|
||||
|
||||
Nothing live is serialized (no ffmpeg process, context, channels, or log sink).
|
||||
This is the contribution of PR #174 ("RC"), which also unified the native and
|
||||
Jellyfin-compat paths behind one `playback.TranscodeManager` so the
|
||||
card-lifetime rules, the reconstruct concurrency cap, and the node-affinity rule
|
||||
live in exactly one place. The reconstruct skeleton —
|
||||
`LoadOrReconstructSession`, `ReconstructSession` / `ReconstructTranscode`, the
|
||||
single-flight + `reconstructSem` cap that paces the post-restart thundering
|
||||
herd, `RegisterReconstructed`, the `CloseProcess`/`Close` split — is the
|
||||
foundation every later option reused. **It was kept; only the descriptor storage
|
||||
and the revocation model evolved.**
|
||||
|
||||
The six routes that must survive a restart:
|
||||
|
||||
| # | Route | Tier | How it is rebuilt |
|
||||
|---|-------|:----:|-------------------|
|
||||
| 1 | native direct | 1 | client `Range` re-serve; descriptor = identity |
|
||||
| 2 | native remux | 1 | re-spawn ffmpeg at `?seek`; descriptor = id + audio |
|
||||
| 3 | native transcode | 2 | rebuild `Session` + ffmpeg seeked to the requested segment; descriptor = full encode opts |
|
||||
| 4 | jellycompat direct | 1 | compat store + `Range` re-serve |
|
||||
| 5 | jellycompat remux | 1 | compat store + `?seek` |
|
||||
| 6 | jellycompat transcode | 2 | compat store + shared-manager reconstruct |
|
||||
|
||||
Tier 1 = stateless re-serve (client re-supplies position). Tier 2 = rebuild
|
||||
ffmpeg. Routes 4–6 always retain a durable central compat store
|
||||
(`jellycompat_playback_sessions`): third-party Jellyfin clients cannot
|
||||
round-trip a native descriptor, so the compat layer keeps the full
|
||||
`PlaybackSession` (route + media source resolved before native reconstruct
|
||||
runs).
|
||||
|
||||
---
|
||||
|
||||
## 3. The two axes the options differ on
|
||||
|
||||
Every option is a point on a 2-D grid: **where the descriptor lives** (vertical)
|
||||
× **how revocation is bounded** (horizontal).
|
||||
|
||||
```text
|
||||
AUTHORITY / REVOCATION MODEL ----------------------->
|
||||
re-resolve at long token short TTL +
|
||||
reconstruct (24h, expire) refresh checkpoint
|
||||
D central +----------------+----------------+----------------+
|
||||
E Postgres row | RC (#174) | (n/a) | RC-async |
|
||||
S +----------------+----------------+----------------+
|
||||
C client/node- | (n/a) | TR-24h | TR + re-mint |
|
||||
R carried token | | | |
|
||||
I +----------------+----------------+----------------+
|
||||
P
|
||||
T top row = descriptor stays central (node serves bytes but cannot
|
||||
| self-reconstruct → node-restart gap)
|
||||
| bottom = recipe travels to the node (node self-sufficient, no central record)
|
||||
```
|
||||
|
||||
**TR-lease is a fourth point off this grid:** bottom row (token-carried, 24h),
|
||||
but the revocation column is replaced by a *server-side deny marker* — on an
|
||||
admin kill, central would write a per-session deny to Redis that the node
|
||||
enforces by withholding bytes. Revocation is decoupled from the token TTL and
|
||||
needs no client refresh. (An earlier iteration polled and re-resolved every
|
||||
stream on a timer; that was cut to the event-driven session-deny — see §7 and
|
||||
the §8 design note.)
|
||||
|
||||
> **Status: deferred to a future PR.** The deny-marker column described above is
|
||||
> the design target, not the current shipped behavior. As shipped, the
|
||||
> token-carried row offers no revocation tighter than the 24h token TTL on the
|
||||
> node path; admin Terminate and user Stop only tear down the live session and
|
||||
> ffmpeg producer.
|
||||
|
||||
---
|
||||
|
||||
## 4. Goals weighed
|
||||
|
||||
| ID | Goal |
|
||||
|----|------|
|
||||
| G1 | Hot path (per-segment serve) stays off central/Postgres |
|
||||
| G2 | Bytes offloaded to nodes (served from disk, signature-verified, never central) |
|
||||
| G3 | Restart resiliency across all 6 routes |
|
||||
| G4 | Two-factor ownership (auth caller + ownership rebind; refuse `userID==0`/mismatch) |
|
||||
| G5 | Revocation latency (how fast a banned user / pulled access stops playing) |
|
||||
| G6 | Single-box: zero external store required for reconstruction |
|
||||
| G7 | Restart runway (descriptor stays valid longer than the outage) |
|
||||
| G8 | Multi-front-end / verify-anywhere (no integrated-transcode split-brain) |
|
||||
| G9 | Observability — durable, identity-rich active-stream record |
|
||||
| G10 | Low operational surface (tables, migrations, stores, endpoints) |
|
||||
| G11 | Node-restart recovery |
|
||||
| G12 | No new client coordination required |
|
||||
| G13 | Small signing-key blast radius |
|
||||
| G14 | Cleanup correctness (never wipe a live session's segment dir) |
|
||||
|
||||
A note on G9 (observability): the *live* "who is streaming now" view does **not**
|
||||
depend on the descriptor strategy. Every offload node calls the Redis-backed
|
||||
node-session tracker (`internal/nodesessions/tracker.go`) on every serve, tied to
|
||||
actual byte-serving (so a non-cooperative client or replayed token still shows
|
||||
up) and surviving a central restart. G9 scores only the *durable, identity-rich*
|
||||
(who + what + recipe) slice on top of that.
|
||||
|
||||
---
|
||||
|
||||
## 5. The options, and the comparison
|
||||
|
||||
Legend: ✓ meets · ✓\* minor asterisk · ~ partial · ✗ fails · ✓✓ best-in-class
|
||||
|
||||
| Goal | `main` | RC #174 | TR-24h | TR + re-mint | RC-async | **TR-lease** |
|
||||
|------|:------:|:-------:|:------:|:------------:|:--------:|:------------:|
|
||||
| G1 hot path free of central DB | ✓ | ✓\* | ✓ | ✓ | ✓ | ✓ |
|
||||
| G2 bytes offloaded | ✓ | ✓ | ✓✓ | ✓✓ | ✓✓ | ✓✓ |
|
||||
| G3 restart resiliency (6/6) | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| G4 two-factor ownership | ✓ | ✓ | ~ | ~ | ~ | ✓~ |
|
||||
| G5 revocation latency | ~ | ~ | ✗ (≤24h) | ✓ (≤TTL) | ✓✓ | ✓✓ |
|
||||
| G6 single-box, no external store | ✓ | ✗ | ✓✓ | ~ | ~ | ~ |
|
||||
| G7 restart runway | n/a | ✓ (30m) | ✓✓ (24h) | ~ | ~ | ✓✓ (24h) |
|
||||
| G8 multi-front-end | ✗ | ~ | ✓~ | ✓~ | ✓~ | ✓~ |
|
||||
| G9 durable identity-rich record | ✗ | ✓✓ | ✗ | ✗ | ✓ | ~ |
|
||||
| G10 low operational surface | ✓✓ | ~ | ✓ | ~ | ✗ | ~ |
|
||||
| G11 node-restart recovery | ✗ | ✗ | ✓ | ✓ | ✓ | ✓ |
|
||||
| G12 no new client coordination | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ |
|
||||
| G13 small signing-key blast radius | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ |
|
||||
| G14 cleanup correctness | ✓ | ✓✓ | ~ | ~ | ~ | ~ |
|
||||
|
||||
The TTL/revocation strategies head-to-head. **Note: the session-deny column is
|
||||
the deferred design target — see §7; as shipped, the node path has no revocation
|
||||
tighter than the 24h token TTL.**
|
||||
|
||||
| Strategy | Revocation latency | Restart runway | Client refresh? | Central on refresh? |
|
||||
|----------|--------------------|----------------|-----------------|---------------------|
|
||||
| Re-resolve at reconstruct (RC) | on reconstruct; node hop ≤24h | 30 min | no | only on reconstruct |
|
||||
| Leave 24h, expire only (TR-24h) | up to 24h | 24h | no | never |
|
||||
| Short TTL + refresh + denylist (RC-async / TR item B) | ≤~5 min | ~90 s grace | **yes** | every ~3.5 min/session |
|
||||
| 24h token + session-deny marker (TR-lease — **deferred**) | *design target:* admin kill cuts node bytes on next serve; passive ban = next-play / ≤24h. *As shipped:* node path bounded by ≤24h token only | 24h | **no** | nothing on the serve path; one write per admin-kill event |
|
||||
|
||||
---
|
||||
|
||||
## 6. Why each non-final option was set aside
|
||||
|
||||
- **RC (PR #174, the recipe card in Postgres).** Correct and the basis of
|
||||
everything here, but it couples reconstruction to a shared per-session Postgres
|
||||
store: a write on every start, per audio/quality change, and a ≤1/min refresh,
|
||||
with resilience tied to the DB being reachable. It does not give the single-box
|
||||
zero-dependency property, and the node hop's 24h token stays unrevocable
|
||||
anyway. RC's reconstruct *core* was kept; its *storage* was the thing to evolve.
|
||||
|
||||
- **TR-24h (token-carried, leave the token to expire).** Wins the single-box case
|
||||
(zero external dependency) and removes the per-start DB write, but revocation is
|
||||
bounded by *nothing tighter than the 24h token TTL* on the node path — a banned
|
||||
user keeps node-served direct/remux for up to 24h. This is actually the
|
||||
*status quo* for the node path even pre-branch (the proxy verifies by signature
|
||||
alone, no DB), so TR-24h only adds identity claims to the token without
|
||||
improving revocation. Not acceptable as the endpoint.
|
||||
|
||||
- **TR + re-mint (short TTL + authenticated playlist refresh + denylist), and
|
||||
RC-async (the same revocation idea but keeping an async-written Postgres mirror
|
||||
for observability).** Both bound revocation to ~5 min by having the client
|
||||
refetch a stable `RequireAuth` playlist that re-checks access and re-mints
|
||||
short-lived segment URLs. The cost: a **hard client-coordination requirement**
|
||||
(silo-android / silo-apple / Jellyfin clients must implement the refresh flow),
|
||||
a shrunken restart runway (~90 s grace instead of 30 min — if central is down
|
||||
past the grace, playback stalls), and a new authenticated re-mint transport
|
||||
that does not exist on the supported proxy/node topology. RC-async additionally
|
||||
keeps a write pipeline + denylist + the central row (the highest operational
|
||||
surface of all). They solved revocation by pushing work onto the client and
|
||||
shortening the runway — the opposite of the reliability priorities.
|
||||
|
||||
The open item across TR and RC-async was always **revocation enforcement** ("item
|
||||
B"): bounding revocation needs *either* a per-segment token check on the hot path
|
||||
*or* an authenticated re-mint checkpoint the client drives. TR-lease removes that
|
||||
dilemma.
|
||||
|
||||
---
|
||||
|
||||
## 7. TR-lease: the chosen design
|
||||
|
||||
TR-lease keeps TR's 24h token (so the restart runway is untouched and no client
|
||||
refresh flow is needed) and adds a server-side **session-deny marker** for the
|
||||
one revocation case the node path cannot otherwise cover — no client
|
||||
participation, no steady-state cost.
|
||||
|
||||
**Reconstruction (token-carried).** The signed stream token *is* the durable
|
||||
descriptor. Its claims carry the full byte-affecting recipe plus `uid/pid/mfid`
|
||||
ownership lookup keys. A front-end that lost its in-memory session rebuilds
|
||||
ffmpeg from the token the client re-presents — no shared per-session store, zero
|
||||
external dependency for reconstruction on a single box. `HWAccel`/`HWDevice` are
|
||||
deliberately *not* carried; they are re-resolved from live config so an operator
|
||||
config change applies to reconstructed sessions too. Ownership re-binds to the
|
||||
live authenticated caller and refuses `userID==0` / mismatch — the token's `uid`
|
||||
is never trusted alone (a leaked URL is useless without the owner's auth session
|
||||
on the native path).
|
||||
|
||||
**Revocation (session-deny marker).**
|
||||
|
||||
> **Status: deferred to a future PR.** Everything in this subsection (the
|
||||
> `silo:streamauth:<sid>` deny marker, the admin Stop/Terminate deny write, and
|
||||
> the node-side `Allowed()` enforcement) describes the design target and is **not
|
||||
> present in the current implementation**. As shipped, admin Terminate and user
|
||||
> Stop tear down the live in-memory session and the ffmpeg producer, but a
|
||||
> still-valid stream token can reconstruct the session until its 24h TTL expires;
|
||||
> there is no node-side byte-withholding and no guaranteed/instant revocation on
|
||||
> the node path. The design below is retained for when revocation lands.
|
||||
|
||||
The revocation surface was deliberately
|
||||
kept to the *one* case the offloaded topology cannot otherwise cover, rather than
|
||||
a periodic re-check of every stream. An admin Stop/Terminate would write
|
||||
`silo:streamauth:<sid> = deny` (TTL ≥ the 24h token lifetime); the offload node
|
||||
would read it (a node-local Redis GET, sub-ms) before serving any bytes and
|
||||
return `403` with no bytes on a hit. Enforcement is **byte-withholding, not
|
||||
client cooperation** — a revoked client that ignores the 403 keeps hitting a
|
||||
wall.
|
||||
|
||||
Why this would be sufficient, and what it deliberately does *not* do:
|
||||
|
||||
- **New playback is already blocked at central** — `/playback/start` is
|
||||
`RequireAuth` + access check, so a ban/scope change always takes effect on the
|
||||
next play, everywhere, with no marker needed.
|
||||
- **Cooperative clients already stop** — central holds the realtime WebSocket and
|
||||
can drop it + send a stop on a ban.
|
||||
- **The only thing needing a hard node-side cut** is the *explicit, session-scoped
|
||||
admin kill* of a *non-cooperative* client holding a valid token on a
|
||||
node-served direct/remux stream (no producer to kill). That is exactly what the
|
||||
(deferred) session-deny marker would cover; until it lands this case is bounded
|
||||
by the ≤24h token like the passive bans below.
|
||||
- **Not covered (accepted limitation):** a *passive* ban or *partial* access
|
||||
change (lost one library, lowered rating) does NOT hard-revoke an
|
||||
already-running, non-cooperative node stream. It is enforced at next play and
|
||||
otherwise bounded by the ≤24h token. Closing this would require either a
|
||||
periodic re-resolver (a poll over every active stream — rejected as constant
|
||||
DB load for a rare case) or per-content deny enumeration; both were judged not
|
||||
worth the machinery for a self-hosted server. A user-scoped deny was rejected
|
||||
too: it over-blocks content the viewer still legitimately has access to.
|
||||
|
||||
**Fail-open on absence (deferred).** In the design, the normal case is *no*
|
||||
marker, which serves; a Redis error also serves; only a present deny withholds
|
||||
bytes. This is a deliberate reliability-first choice: availability over
|
||||
revocation latency.
|
||||
|
||||
**Why this shape.** It keeps token-carried reconstruction's single-box and
|
||||
verify-anywhere properties and would add revocation with no client coordination
|
||||
(G12) and essentially no steady-state cost — the marker is written only on the
|
||||
rare admin-kill event, never on a timer. It is, in effect, **TR-24 plus a single
|
||||
session-scoped kill switch**. The kill switch is **deferred**; what shipped is
|
||||
TR-24 (token-carried reconstruction, ≤24h token, no node-side revocation).
|
||||
|
||||
---
|
||||
|
||||
## 8. What was implemented, and where
|
||||
|
||||
This PR ships **Commit 1 only** (token-carried reconstruction). **Commit 2 (the
|
||||
node session-deny marker) is deferred to a future PR** — the `internal/streamauth`
|
||||
package, the proxy `Allowed()` guard, and the admin deny write described below
|
||||
are **not present in the current implementation**.
|
||||
|
||||
**Commit 1 — token-carried reconstruction; retire `transcode_recipes`. (shipped)**
|
||||
- `streamtoken.Claims` gains the recipe + `uid/pid/mfid`
|
||||
(`internal/streamtoken/token.go`).
|
||||
- `playback.RecipeCard` projects to/from claims (`ToClaims` /
|
||||
`RecipeCardFromClaims`); the `Reconstruct*` front door takes the decoded recipe
|
||||
instead of reading Postgres (`internal/playback/recipecard.go`,
|
||||
`transcode_manager.go`).
|
||||
- Native serve URLs carry the token as `?st=`; the manifest rewriter already
|
||||
appends `RawQuery` to segment URIs, so segments inherit it with no route change
|
||||
(`internal/api/handlers/playback.go`, `stream.go`). The integrated server is
|
||||
hit directly, so the query-stripping-proxy concern that motivates a path
|
||||
segment does not apply there; the proxy/node path keeps the token in the URL
|
||||
path as before.
|
||||
- Jellycompat folds the recipe into its durable compat store
|
||||
(`PlaybackSession.Recipe`) since Jellyfin clients cannot round-trip a native
|
||||
token (`internal/jellycompat/...`).
|
||||
- Segment-dir cleanup switches from the card index to in-memory liveness (live
|
||||
map + in-flight reconstruct set + age past the max token TTL); a dedicated node
|
||||
keeps its boot-time full wipe (`internal/playback/transcode_cleanup.go`,
|
||||
`transcode_manager.go`).
|
||||
- `PostgresRecipeStore`, the `RecipeStore` interface, and the unmerged
|
||||
`transcode_recipes` migration are deleted.
|
||||
|
||||
**Commit 2 — node session-deny marker. (DEFERRED — not in this PR)**
|
||||
|
||||
> The items below describe the planned revocation commit. They were removed from
|
||||
> this PR and deferred to a future one; none of these symbols exist in the
|
||||
> current implementation. The admin Stop/Terminate path still tears down the live
|
||||
> session and ffmpeg producer, but writes no deny marker, so a valid token can
|
||||
> reconstruct until its 24h TTL expires.
|
||||
|
||||
- `internal/streamauth`: a deny-only Redis `Store` — `Deny(sid)` (TTL ≥ token
|
||||
lifetime) and a fail-open `Allowed(sid)` reader. No poller, no allow-leases, no
|
||||
access re-resolver.
|
||||
- The node guard in `internal/proxy/server.go` (`verifyToken` consults
|
||||
`Allowed`).
|
||||
- An admin Stop/Terminate writes the deny marker
|
||||
(`internal/api/handlers/admin_playback_control.go` → `PlaybackHandler.LeaseDenier`).
|
||||
- `nodesessions.SessionInfo` carries the numeric ownership keys
|
||||
(`auth_user_id`/`profile_id`/`media_file_id`), populated by the node from the
|
||||
verified token, to enrich the live admin "active streams" view (who/what, not
|
||||
just session id) — see §10 monitoring.
|
||||
- A single integrated box reads no marker and needs none — the producer-kill +
|
||||
session removal on Terminate is the stop there.
|
||||
|
||||
> Design note (about the deferred commit 2): an earlier iteration added a central
|
||||
> *revalidator* that re-resolved access for every active stream every ~2 min and
|
||||
> wrote allow/deny leases. It was dropped in favour of the event-driven
|
||||
> session-deny above: the poll re-derived, at a constant all-streams DB cost,
|
||||
> signals the
|
||||
> system already emits (new playback is gated at `/playback/start`; admin kill is
|
||||
> an explicit event), and its only unique coverage — sub-24h *passive* revocation
|
||||
> of an in-flight node stream — was judged not worth the machinery (see §7).
|
||||
|
||||
---
|
||||
|
||||
## 9. Issues raised across review, and their disposition
|
||||
|
||||
From two adversarial review rounds on the reconstruct core and the storage
|
||||
evolution:
|
||||
|
||||
- **Integrated-transcode split-brain across front-ends (P-1 / A).** The
|
||||
`SessionManager` + reconstruct single-flight are per-process and the descriptor
|
||||
carries no owning-node id, so without sticky affinity two front-ends can run
|
||||
divergent ffmpeg. **Disposition: deferred (owner-identity claim).** Supported
|
||||
topology is dedicated `--mode=transcode` nodes (routed via `tnode`, no
|
||||
split-brain) + single-box / LB-session-affinity integrated transcode. The
|
||||
caveat is documented, not silently assumed.
|
||||
- **24h node-path token is unrevocable (P-2).** The (deferred, see §7–§8)
|
||||
session-deny marker would let the explicit admin kill cut node bytes on the next
|
||||
serve. **Until that ships, this remains open:** the node-path token is
|
||||
unrevocable, so admin kill and passive bans / partial access changes are all
|
||||
bounded by next-play + the ≤24h token.
|
||||
- **Dedicated transcode-node restart (P-3 / tr-10). Fixed for both paths.**
|
||||
*Native:* the proxy forwards the verified stream token to the transcode node
|
||||
(`X-Silo-Stream-Token`); on a manifest/segment miss the node re-verifies the
|
||||
token, decodes the recipe, and self-reconstructs ffmpeg seeked to the requested
|
||||
segment — single-flighted and concurrency-capped exactly like the integrated
|
||||
reconstruct. *Jellycompat:* the node-hop token is server-minted and *could* carry the
|
||||
recipe like native, but it deliberately does not — the recipe is mutated in
|
||||
place under a stable session id (a Jellyfin audio/subtitle switch can arrive as
|
||||
a `/Sessions/Playing/Progress` report that restarts ffmpeg without re-minting
|
||||
the client's token), and a third-party Jellyfin client cannot be driven to
|
||||
refresh a stale token, so a token *snapshot* could reconstruct a stale
|
||||
rendition. Central instead writes the recipe to a shared Redis recipe store
|
||||
(`internal/noderecipe`) keyed by upstream session id and overwrites it on every
|
||||
switch, and the node (which cannot reach Postgres) reads the *current* recipe on
|
||||
the reconstruct miss — over the same Redis the session tracker already uses (and
|
||||
the deferred deny-lease would use), so no central URL is needed. The boot-time
|
||||
full segment-dir wipe
|
||||
stands; reconstruct re-transcodes from the requested segment. See §10 for the
|
||||
full rationale (and why this is *not* a token-carryable case).
|
||||
- **`userID==0` tolerance / "auth optional" (P-5).** Reconstruct hard-rejects
|
||||
`userID==0` and a card whose session id does not match the URL.
|
||||
- **Mutable recipe under a stable id (D).** A parameter change re-mints a new
|
||||
manifest + token and the client reloads, so the common case self-heals. The
|
||||
only residual is a restart *during* the brief switch transition with a prior
|
||||
token in flight: it reconstructs the prior *rendition* (access is re-resolved,
|
||||
only the rendition can be stale). TTL-bounded, self-healing on reload; minor,
|
||||
accepted.
|
||||
- **Cleanup liveness (F / tr-9).** Each process owns its `TranscodeDir`, so the
|
||||
liveness signal is the in-process live map + the in-flight reconstruct set +
|
||||
age-past-max-TTL; a dir is reaped only when absent from both and older than any
|
||||
surviving token. No cross-process enumeration-failure mode for the supported
|
||||
topology.
|
||||
- **Signing-key blast radius (E / tr-7, G13).** A single `JWTSecret` signs native
|
||||
auth, the stream token, and the node bearer, so a node-secret leak can forge
|
||||
native logins. **Accepted risk** on the trusted-node precondition. **Deferred
|
||||
hardening:** split the stream-token secret from the native-auth secret, then
|
||||
asymmetric stream-token signing before any untrusted/edge node.
|
||||
|
||||
---
|
||||
|
||||
## 10. Residuals and follow-ups
|
||||
|
||||
- **Revocation of an in-flight node stream (all cases, until deny-lease ships).**
|
||||
Because the session-deny marker is deferred (§7–§8), *no* ban — admin kill,
|
||||
passive ban, or partial access change — hard-cuts an already-running,
|
||||
non-cooperative node-served stream; all are enforced at next play, bounded by
|
||||
the ≤24h token. The future deny-lease closes the *admin-kill* case; passive
|
||||
bans would still need a per-content deny enumeration on the access-change event,
|
||||
or accepting it. See §7.
|
||||
- **Monitoring.** Two distinct questions:
|
||||
- *Live "who is watching what right now"* — covered. Every node serve writes a
|
||||
record to the Redis tracker (`silo:sessions:*`), now enriched with
|
||||
`auth_user_id` / `media_file_id` from the verified token, so the admin
|
||||
active-streams view answers who/what/where, tied to real byte-serving and
|
||||
surviving a central restart. Integrated (no-node) sessions show in central's
|
||||
in-memory session list.
|
||||
- *Durable historical/audit ("who watched what last week")* — NOT provided by
|
||||
this subsystem (there is no longer a durable per-session recipe record, by
|
||||
design — G9 was the deliberate trade for token-carried reconstruction). That
|
||||
history lives in the separate watch-progress / scrobble system.
|
||||
- **Multiple viewers on one stream token.** The token is per playback session, so
|
||||
a shared/leaked stream URL lets several clients pull bytes under one session id.
|
||||
They **collapse to a single entry** in the tracker (one `sid`) — monitoring
|
||||
cannot distinguish or count them — and the (deferred) session-deny would cut
|
||||
them as a group, not individually. Detecting/limiting this needs a per-account
|
||||
concurrency cap or
|
||||
per-device session ids (device binding); deferred. Do not bind to client IP
|
||||
(mobile / CGNAT).
|
||||
- **Token in logs (tr-1).** Scrub the `?st=` token / path token from request
|
||||
loggers (the jellycompat logger logs `RawQuery`; the native logger omits it).
|
||||
Short TTL + signature bound the exposure.
|
||||
- **Owner-identity (A).** Required before multi-front-end *integrated* transcode
|
||||
without sticky affinity; defines owner election, self-URL discovery, a
|
||||
peer-forward endpoint, and failover.
|
||||
- **Node-side reconstruct (P-3). Done** for native (token-forwarded) and
|
||||
jellycompat (Redis recipe handoff via `internal/noderecipe`: central writes the
|
||||
recipe at remote-transcode start, the node reads it on a reconstruct miss).
|
||||
Precondition: central and the nodes share the same Redis (already required for
|
||||
the session tracker, and for the deferred deny-lease). Residual: the recipe
|
||||
store is consulted
|
||||
only for a jellycompat token, so a node restart still depends on the recipe key
|
||||
surviving (24h TTL, ≥ token lifetime).
|
||||
- *Why a server-side store and not the token here (the real rationale).* It is
|
||||
tempting to delete `internal/noderecipe` and let the jellycompat node-hop
|
||||
token carry the recipe like native — the token is server-minted and opaque to
|
||||
the Jellyfin client (it follows it only as a redirect target), so it *could*.
|
||||
That does **not** work, and the reason is **not** the often-stated "a Jellyfin
|
||||
client can't round-trip a token." The reason is that the recipe is **mutated
|
||||
in place under a stable session id**: a Jellyfin audio/subtitle switch can
|
||||
arrive as a `/Sessions/Playing/Progress` report
|
||||
(`restartCompatTranscodeForAudioSelection`) that restarts ffmpeg in place
|
||||
**without re-minting the client's token**. A third-party Jellyfin client
|
||||
cannot be driven to refresh the now-stale token — its VOD/`ENDLIST` manifest
|
||||
is fetched once, and the compat WebSocket (`HandleSocket`) is a receive-only
|
||||
keep-alive stub with no server→client command to force a manifest re-fetch. A
|
||||
token is a snapshot at mint time, so after such a switch it would reconstruct
|
||||
the **stale** rendition (old audio/subtitle) until the client happens to
|
||||
re-fetch the manifest (e.g. on a seek), which for a VOD stream may be never.
|
||||
Because the node also cannot reach Postgres (where the authoritative compat
|
||||
recipe lives), the only way the node rebuilds the *current* recipe is a
|
||||
node-reachable, server-authoritative store that central overwrites on every
|
||||
switch — i.e. `internal/noderecipe`. Native has no equivalent path: every
|
||||
native audio switch re-mints the manifest + token, so the native client's
|
||||
token is never stale, which is exactly why native needs no such store. The
|
||||
store is **load-bearing, not redundant**.
|
||||
- **Key split / asymmetric signing (E).** Closes the account-takeover path; a
|
||||
precondition for untrusted edge nodes.
|
||||
|
||||
---
|
||||
|
||||
## 11. Preconditions
|
||||
|
||||
- **Persistent `TranscodeDir` (recommended).** With persistence, reconstruct
|
||||
serves surviving segments; with an ephemeral dir it re-transcodes from the
|
||||
descriptor (which survives a wiped dir). A node *restart* still loses the
|
||||
session regardless (P-3).
|
||||
- **Re-transcode of one segment beats the buffer runway** on worst-case hardware
|
||||
(software decode, subtitle burn-in). Measure.
|
||||
- **Client retry/backoff exists.** Clients retry segment/manifest GETs on
|
||||
5xx/connection error and reconnect the WebSocket. The token rides the URL, so no
|
||||
extra client logic is needed to carry it; a client that does not retry sees the
|
||||
failed request as fatal.
|
||||
|
||||
---
|
||||
|
||||
## 12. Diagrams
|
||||
|
||||
The `mermaid` blocks render in GitHub / VS Code / most Markdown viewers; the
|
||||
ASCII blocks render anywhere.
|
||||
|
||||
### 12.1 Topology — where state lives
|
||||
|
||||
> **Deferred:** the `silo:streamauth:<sid> = deny` marker, the "Admin
|
||||
> Stop/Terminate → write session-deny marker" arrow, and the node's "deny GET"
|
||||
> are the planned revocation path and are **not** in the current implementation.
|
||||
> The tracker (`silo:sessions:*`) and the recipe handoff are shipped.
|
||||
|
||||
```text
|
||||
┌─────────────────────────── CENTRAL (mode=server) ───────────────────────────┐
|
||||
Client │ RequireAuth /playback/start /playback/transcode/start (mint token) │
|
||||
(web / android / │ native serve: /playback/transcode/{sid}/master.m3u8?st= /stream/{sid}?st= │
|
||||
apple / jellyfin) │ Admin Stop/Terminate → write session-deny marker (event, not a timer) │
|
||||
└───────────┬──────────────────────────┬───────────────────────┬──────────────┘
|
||||
│ Postgres │ Redis │
|
||||
▼ ▼ │ (multi-node only)
|
||||
┌───────────────────────┐ ┌──────────────────────────┐ ▼
|
||||
│ catalog / access │ │ silo:sessions:* (tracker)│ ┌────────────────────────┐
|
||||
│ jellycompat_playback_ │ │ silo:streamauth:<sid> │ │ Proxy node (mode=proxy) │
|
||||
│ sessions (compat │ │ = deny (session kill) │◀─│ verifyToken + deny GET │
|
||||
│ store, carries Recipe)│ └──────────────────────────┘ │ serves direct/remux │
|
||||
└───────────────────────┘ ▲ │ proxies transcode ──────┼──▶ Transcode node
|
||||
✗ NO transcode_recipes table (deleted this branch) │ node-local GET └────────────────────────┘ (mode=transcode)
|
||||
└── sub-ms, off Postgres ffmpeg /transcode/{sid}/…
|
||||
```
|
||||
|
||||
Descriptor location by route (the core change). The "lease (node)" revocation
|
||||
column is the **deferred** design; as shipped, the node path has no revocation
|
||||
tighter than the ≤24h token:
|
||||
|
||||
| route family | descriptor (how it reconstructs) | revocation |
|
||||
|--------------|----------------------------------|------------|
|
||||
| native (1–3) | the signed token the client re-presents | *deferred:* lease (node) / shipped: producer-kill (integrated), ≤24h token (node) |
|
||||
| jellycompat (4–6) | `jellycompat_playback_sessions.data.Recipe` (token can't round-trip) | *deferred:* lease (node) / shipped: producer-kill, ≤24h token (node) |
|
||||
| ~~all~~ | ~~`transcode_recipes` Postgres row~~ — removed | — |
|
||||
|
||||
### 12.2 Where the token rides
|
||||
|
||||
```text
|
||||
NATIVE (integrated server is hit directly → query param; segments inherit it)
|
||||
manifest : GET /playback/transcode/{sid}/master.m3u8?st=<JWT>
|
||||
segment : GET /playback/transcode/{sid}/segment/seg_00042.m4s?st=<JWT> ← RawQuery propagated by the manifest rewriter
|
||||
direct : GET /stream/{sid}?st=<JWT>
|
||||
|
||||
PROXY/NODE (capability URL → path segment; survives a query-stripping hop)
|
||||
GET {proxy}/stream/transcode/{JWT}/master.m3u8 → relative segment URIs inherit {JWT}
|
||||
GET {proxy}/stream/direct/{JWT} GET {proxy}/stream/remux/{JWT}
|
||||
|
||||
JELLYCOMPAT → recipe is NOT in the token; it lives in the compat store.
|
||||
the path token is used only for the node hop.
|
||||
```
|
||||
|
||||
### 12.3 Normal playback — native, integrated box (no nodes)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server (integrated)
|
||||
participant PG as Postgres (access)
|
||||
Note over C,S: transcode start
|
||||
C->>S: POST /playback/transcode/start (RequireAuth)
|
||||
S->>PG: resolve file + access
|
||||
S->>S: StartTranscode (ffmpeg) + RegisterTranscodeSession
|
||||
S->>S: card = NewRecipeCard(opts); token = Sign(card.ToClaims(), 24h)
|
||||
S-->>C: 202 manifestURL = /playback/transcode/{sid}/master.m3u8?st=TOKEN
|
||||
Note over C,S: hot path — NO Postgres, NO lease (live session in memory)
|
||||
C->>S: GET .../master.m3u8?st=TOKEN
|
||||
S->>S: LoadOrReconstructSession → live hit → BuildPlaybackManifest (segments carry ?st)
|
||||
S-->>C: manifest
|
||||
loop every segment
|
||||
C->>S: GET .../segment/seg_N.m4s?st=TOKEN
|
||||
S->>S: live session → GetSegment → ServeFile
|
||||
S-->>C: bytes
|
||||
end
|
||||
```
|
||||
|
||||
The hot path touches neither Postgres nor the lease — it is an in-memory map
|
||||
hit. `?st` is verified only to decode a card if reconstruct is needed; a live
|
||||
session ignores it.
|
||||
|
||||
### 12.4 Normal playback — native, multi-node (offloaded)
|
||||
|
||||
> **Deferred:** the `GET silo:streamauth:{sid}` lease guard and its `deny → 403`
|
||||
> branch are the planned revocation path, **not** in the current implementation.
|
||||
> As shipped the node verifies the token, tracks the session, and serves — there
|
||||
> is no deny check.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server
|
||||
participant N as Proxy node
|
||||
participant T as Transcode node
|
||||
participant R as Redis
|
||||
C->>S: POST /playback/transcode/start (RequireAuth)
|
||||
S->>S: card(full recipe, tnode=T); token = Sign(card.ToClaims())
|
||||
S-->>C: manifestURL = {N}/stream/transcode/TOKEN/master.m3u8
|
||||
loop manifest + segments
|
||||
C->>N: GET /stream/transcode/TOKEN/...
|
||||
N->>N: verifyToken(TOKEN)
|
||||
N->>R: GET silo:streamauth:{sid} (lease guard)
|
||||
alt deny
|
||||
N-->>C: 403 (no bytes)
|
||||
else allow / absent
|
||||
N->>R: Track silo:sessions:{node}:{sid} (uid, mfid from claims)
|
||||
N->>T: proxy /transcode/{sid}/...
|
||||
T-->>N: bytes
|
||||
N-->>C: bytes
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
The node copies `uid`/`mfid` from the verified token into the Redis tracker —
|
||||
that is what enriches the live admin "active streams" view (who/what per node).
|
||||
|
||||
### 12.5 Restart → reconstruct (native transcode)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as Client
|
||||
participant S as Central server (just restarted — session map empty)
|
||||
Note over S: in-memory session GONE; NO transcode_recipes table to read
|
||||
C->>S: GET /playback/transcode/{sid}/segment/seg_42.m4s?st=TOKEN
|
||||
S->>S: card = streamCardFromQuery(?st) → Verify + decode recipe
|
||||
S->>S: LoadOrReconstructSession → GetSession = NotFound, card != nil
|
||||
S->>S: ReconstructSession (re-bind: refuse userID==0 / != card.uid; card.sid == {sid})
|
||||
S->>S: ReconstructTranscode(seg=42) → single-flight, mark in-flight, respawn ffmpeg SEEKED to seg 42
|
||||
S-->>C: bytes (resumes near where the client was)
|
||||
```
|
||||
|
||||
- direct/remux restart is the same but Tier-1: reconstruct the `Session` from the
|
||||
identity token, then `Range` / `?seek` re-serve — no ffmpeg rebuild.
|
||||
- multi-node restart: the integrated front-end reconstructs only the `Session`
|
||||
(to learn `tnode`) and re-proxies; the transcode node keeps serving. If the
|
||||
transcode *node* itself reboots, the proxy forwards the stream token
|
||||
(`X-Silo-Stream-Token`) and the node self-reconstructs ffmpeg seeked to the
|
||||
requested segment (native path, P-3). A jellycompat session on a rebooted node
|
||||
reconstructs too: the node fetches the recipe central wrote to the shared Redis
|
||||
recipe store (`internal/noderecipe`) at transcode start, since the Jellyfin
|
||||
client's token cannot carry it.
|
||||
- cleanup race: a segment dir is reaped only if absent from the live map AND the
|
||||
in-flight set AND older than 24h (max token TTL), so a token that can still
|
||||
reconstruct never gets its dir wiped.
|
||||
|
||||
### 12.6 Auth deny — the session-deny marker
|
||||
|
||||
> **Status: deferred to a future PR.** The entire "Admin Stop/Terminate → write
|
||||
> deny marker" and "Offload node → GET deny → 403" flow below is the planned
|
||||
> revocation design and is **not** in the current implementation. As shipped,
|
||||
> admin Stop/Terminate does the "realtime WS stop + producer teardown" half only
|
||||
> (the right branch); it writes no marker, and the node performs no deny check, so
|
||||
> a valid token reconstructs until its ≤24h TTL. New playback is still gated at
|
||||
> `/playback/start` (the top subgraph is accurate).
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph New["New playback (any topology)"]
|
||||
P["/playback/start — RequireAuth + access check"] --> PB{allowed?}
|
||||
PB -- no --> PR[refused — ban takes effect at next play]
|
||||
PB -- yes --> PM[mint token]
|
||||
end
|
||||
subgraph Admin["Admin Stop/Terminate (event, not a timer)"]
|
||||
K[denyStreamLease] --> WD[SET silo:streamauth:sid = deny<br/>TTL = 24h]
|
||||
K --> WS[realtime WS stop + producer teardown]
|
||||
end
|
||||
subgraph Node["Offload node — every serve"]
|
||||
G[verifyToken OK] --> L{GET silo:streamauth:sid}
|
||||
L -- present deny --> F[403, no bytes]
|
||||
L -- absent / redis-err --> SV[serve bytes fail-open]
|
||||
end
|
||||
```
|
||||
|
||||
Invariants:
|
||||
- No steady-state writes: the marker is set only on an admin kill, never on a
|
||||
timer. The normal serve path finds *no* key and serves.
|
||||
- Fail-open: absent key or Redis error → serve. The only hard stop is a present
|
||||
deny.
|
||||
- The integrated box reads no marker — the producer-kill + session removal on
|
||||
Terminate is the stop there; the marker only matters where nodes serve.
|
||||
- Not covered here (by design): a *passive* ban / partial access change of an
|
||||
in-flight non-cooperative node stream — enforced at next play, bounded by the
|
||||
≤24h token (§7).
|
||||
|
||||
### 12.7 Jellycompat — what differs
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant J as Jellyfin client
|
||||
participant S as Central (jellycompat)
|
||||
participant PG as Postgres compat store
|
||||
J->>S: negotiate → PlaybackSession{UpstreamSessionID, MediaSources}
|
||||
S->>PG: Put (jellycompat_playback_sessions)
|
||||
Note over S: transcode start
|
||||
S->>S: StartTranscode; recipe = NewRecipeCard(opts)
|
||||
S->>PG: Update(ps): TranscodeStarted=true, Recipe=recipe (same write)
|
||||
Note over J,S: restart → reconstruct from the COMPAT STORE, not a token
|
||||
J->>S: GET HLS segment
|
||||
S->>PG: Get(playSessionID) → ps.Recipe
|
||||
S->>S: LoadOrReconstructSession(UpstreamSessionID, card = ps.Recipe) → ReconstructTranscode(*ps.Recipe)
|
||||
S-->>J: bytes
|
||||
```
|
||||
|
||||
Jellyfin clients cannot carry a native `?st` token, so the recipe is folded into
|
||||
the durable compat row (`PlaybackSession.Recipe`, persisted in the `data` JSONB)
|
||||
and written in the same `Update` that flips `TranscodeStarted`. The node hop
|
||||
(multi-node jellycompat) still uses a path token exactly like native (the lease
|
||||
guard is deferred along with the rest of the revocation path).
|
||||
|
||||
### 12.8 Hot-path cost per request
|
||||
|
||||
The "lease" / "SET deny" Redis costs below are **deferred** (the revocation path
|
||||
is not shipped); as shipped the node does only the tracker write, and admin
|
||||
Stop/Terminate writes no Redis key:
|
||||
|
||||
| request | Postgres | Redis | central CPU | reconstruct |
|
||||
|---------|----------|-------|-------------|-------------|
|
||||
| native segment, live, integrated | — | — | map hit | — |
|
||||
| native segment, live, node | — | 1 track (+ deferred: GET lease) | — | — |
|
||||
| native segment, after restart, integrated | — | — | verify JWT + ffmpeg respawn (once, single-flight) | yes |
|
||||
| jellycompat segment, live | — | track (+ deferred: lease) | map hit | — |
|
||||
| jellycompat segment, after restart | 1 Get (compat row) | track (+ deferred: lease) | respawn | yes |
|
||||
| admin Stop/Terminate (event, not per-request) | — | deferred: 1 SET deny | one write | — |
|
||||
|
||||
---
|
||||
|
||||
## 13. Restart-resiliency behavior matrix (role × path)
|
||||
|
||||
The observable reconstruction behavior of every playback path under a restart of
|
||||
each server role. **✅** = the path reconstructs and resumes on the client's next
|
||||
request; **—** = the path never executes on that role (nothing to reconstruct);
|
||||
**⚠️** = reconstructs only while a stated precondition holds.
|
||||
|
||||
| path | integrated restart | proxy-node restart | transcode-node restart |
|
||||
|------|--------------------|--------------------|------------------------|
|
||||
| native · direct | ✅ rebuild `Session` from `?st` token → `Range` re-serve | ✅ stateless re-serve from `claims.MediaPath` | — not served on a transcode node |
|
||||
| native · remux | ✅ token + `?seek` → re-spawn pipe | ✅ stateless per-request pipe re-spawn | — |
|
||||
| native · transcode | ✅ token (full recipe) → respawn ffmpeg seeked to segment | ✅ transparent re-route + token forward | ✅ self-contained — token is recipe-complete |
|
||||
| jellycompat · direct | ✅ re-resolve from durable compat row → `Range` re-serve | ✅ stateless re-serve from token `MediaPath` | — |
|
||||
| jellycompat · remux | ✅ compat row + `?seek` | ✅ stateless per-request pipe re-spawn | — |
|
||||
| jellycompat · transcode | ✅ `ReconstructTranscode(*ps.Recipe)` from compat row | ✅ transparent re-route + token forward | ⚠️ reconstructs **iff** the shared Redis recipe (`internal/noderecipe`, ≤24h TTL) still exists; fails closed (404) otherwise |
|
||||
|
||||
**What carries the recovery state, per role**
|
||||
|
||||
| role | native carrier | jellycompat carrier |
|
||||
|------|----------------|---------------------|
|
||||
| integrated | `?st` stream token — identity for direct/remux, full recipe for transcode | durable compat row `PlaybackSession.Recipe` |
|
||||
| proxy node | self-describing path token, re-derived per request — the proxy holds no per-session state | same path token (identity); the recipe is not needed at the proxy |
|
||||
| transcode node | forwarded `X-Silo-Stream-Token`, recipe-complete | identity token + Redis `noderecipe` recipe (the Jellyfin token cannot carry it) |
|
||||
|
||||
**Reading the matrix**
|
||||
|
||||
- **Integrated and proxy-node restarts are unconditional** for every applicable
|
||||
path. The proxy holds no authoritative per-session state — direct/remux are
|
||||
re-served from the token's `MediaPath`, transcode is reverse-proxied — so a
|
||||
proxy bounce, or a reroute to any in-group proxy (`pickProxy` is round-robin
|
||||
with no session affinity), is transparent.
|
||||
- **The four `—` cells are structural, not gaps.** Direct and remux never run on
|
||||
a transcode node — the proxy (or the integrated box) serves them from source
|
||||
media — so a transcode-node restart has nothing to reconstruct on those paths.
|
||||
- **The one conditional (⚠️)** is jellycompat transcode after a *transcode-node*
|
||||
restart. The node-hop token is identity-only — not because a Jellyfin client
|
||||
cannot round-trip it (the token is server-minted and opaque to the client), but
|
||||
because the recipe is mutated in place under a stable session id and a
|
||||
third-party Jellyfin client cannot be driven to refresh a stale token, so a
|
||||
token snapshot could reconstruct a stale rendition (see §10). The node therefore
|
||||
rebuilds from the Redis `noderecipe` entry central overwrites on every switch
|
||||
rather than from the token. Native transcode has no such dependency — every
|
||||
native audio switch re-mints the token, so it is always recipe-complete and
|
||||
current. This is the only native-vs-jellycompat asymmetry in reconstruction
|
||||
(see §10).
|
||||
|
||||
**Out of scope of this matrix** (accepted behaviors, not restart-reconstruction
|
||||
holes — see §7, §10):
|
||||
|
||||
- Passive ban / partial access change of an in-flight node stream — enforced at
|
||||
next play, bounded by the ≤24h token, not by reconstruction.
|
||||
- Integrated transcode behind a non-sticky load balancer can split-brain into
|
||||
divergent segment dirs (P-1); safe single-front-end or with LB affinity. Remote
|
||||
(node) sessions are immune — node affinity routes every front-end to the same
|
||||
transcode node.
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,311 @@
|
||||
# Plan: interleave `transcribe_translate` so live AI subtitles follow the playhead
|
||||
|
||||
Issue: https://github.com/Silo-Server/silo-server/issues/154
|
||||
Area: `internal/subtitles/ai` (+ `internal/playback` extraction, `internal/config`)
|
||||
Type: bug fix to an existing (shipped) capability — makes the chained AI-subtitle
|
||||
path deliver its designed live behavior. No new client-facing contract.
|
||||
|
||||
Commands assume the repository root is the cwd.
|
||||
|
||||
## Problem (confirmed)
|
||||
|
||||
`Service.runTranscribe` (`internal/subtitles/ai/service.go`) runs the chained
|
||||
`transcribe_translate` kind as two **global, sequential** stages:
|
||||
|
||||
1. Transcribe the **entire** file (progress 5%→70%). Transcript cues are not
|
||||
streamed for the chained kind —
|
||||
`streamTranscript := streaming && job.Kind == JobKindTranscribe`
|
||||
(`service.go:417`), so it is `false` for `transcribe_translate`.
|
||||
2. Only then translate **all** cues (70%→95%), streaming each translated batch
|
||||
as it lands (`service.go:482-493`).
|
||||
|
||||
Because translation cannot start until step 1 finishes for the whole file, the
|
||||
first translated cue streams minutes into the job, long past the playhead.
|
||||
|
||||
### Two distinct bottlenecks (both must be addressed)
|
||||
|
||||
- **Transcription stage is global.** Translation waits for the whole-file ASR
|
||||
pass. This is the dominant term for a feature film (~20 min ASR) and the
|
||||
primary cause. Fixed by interleaving translation per chunk (Phase 1).
|
||||
- **Extraction is global and precedes any callback.** `Transcribe` calls
|
||||
`t.extract(...)` (`transcriber.go:124`) before the `onChunk` loop, and
|
||||
`ExtractAudioChunks` blocks on a single whole-track ffmpeg pass
|
||||
(`internal/playback/audio_extract.go:34,65`). So even with per-chunk
|
||||
interleave and small chunks, the first cue still waits for full-file
|
||||
extraction (~1–2 min for a 2h film). Fixed by playhead-first incremental
|
||||
extraction (Phase 2).
|
||||
|
||||
The transcriber already exposes the per-chunk seam: `Transcribe` invokes
|
||||
`onChunk(cues, done, total)` per chunk, playhead-first (`transcriber.go:111-174`),
|
||||
and plain `transcribe` already streams through it (`service.go:426-429`).
|
||||
|
||||
Secondary defect found while confirming: `WhisperTranscriber.SetExtraction`
|
||||
clamps any `chunkSeconds` outside `[60, 600]` to the **default 600**
|
||||
(`transcriber.go:100-102`), so a small live chunk value silently becomes 10 min
|
||||
instead of clamping to the floor.
|
||||
|
||||
## Goal & honest latency targets
|
||||
|
||||
Translated cues stream playhead-first for a session-attached
|
||||
`transcribe_translate` job, instead of waiting for whole-file transcription.
|
||||
|
||||
Time-to-first-translated-cue, by phase:
|
||||
|
||||
- **Today:** `extract(full) + transcribe(full) + translate(first batch)` — many
|
||||
minutes (≈ full ASR pass).
|
||||
- **After Phase 1 (interleave, existing full extraction):**
|
||||
`extract(full) + transcribe(first chunk) + translate(first chunk)` — removes
|
||||
the whole-file ASR pass from the critical path; ≈ extraction time (~1–2 min for
|
||||
a 2h film), then cues follow the playhead. Already usable for live after a
|
||||
one-time wait.
|
||||
- **After Phase 2 (playhead-first incremental extraction):**
|
||||
`extract(one chunk near playhead) + transcribe + translate` — single-digit
|
||||
seconds. This is what the issue means by "within seconds."
|
||||
|
||||
Phase 1 delivers most of the value with low risk; Phase 2 closes the remaining
|
||||
extraction latency. Ship Phase 1 first; do not claim "within seconds" until
|
||||
Phase 2 lands.
|
||||
|
||||
### Non-goals (this plan)
|
||||
|
||||
- Pacing/throttling to the playhead (rolling window). Optional in the issue;
|
||||
follow-up below.
|
||||
- Cross-chunk translation-context continuity beyond batch context neighbours.
|
||||
|
||||
## Decision: interleave only when streaming
|
||||
|
||||
**Session-attached (live) jobs** use the interleaved per-chunk path.
|
||||
**Background (non-streaming) `transcribe_translate` jobs keep the current
|
||||
whole-file transcribe-then-translate-at-end path unchanged.** Rationale: the
|
||||
issue is live-only; this preserves background output byte-for-byte (full
|
||||
cross-file translation context, no incremental-extraction cost) and keeps the
|
||||
two behaviors cleanly separated by the existing `streaming` boolean. The
|
||||
shared transcript-storage step is factored so it is not duplicated.
|
||||
|
||||
---
|
||||
|
||||
## Phase 1 — interleave translation per chunk (streaming jobs)
|
||||
|
||||
### 1a. `internal/subtitles/ai/service.go` — `runTranscribe`
|
||||
|
||||
For the chained kind **when `streaming`**, replace the kind-specific `onChunk`
|
||||
body (`service.go:423-430`) with per-chunk translate-and-stream:
|
||||
|
||||
- Capture a translate error in a closure: `var translateErr error` and an
|
||||
accumulator `var translatedAccum []SubtitleCue`.
|
||||
- In `onChunk(chunk, done, total)`:
|
||||
- If `translateErr != nil` or `len(chunk) == 0`, return (skip silence; stop
|
||||
translating after a prior failure). Do **not** abort transcription.
|
||||
- Else call `s.translator.Translate` on **that chunk's** cues. On error: set
|
||||
`translateErr`, emit `TranslationFailed` once, and return (let transcription
|
||||
continue). On success: stream translated cues via `TranslationCues` and
|
||||
append to `translatedAccum`.
|
||||
- Progress: single 5%→95% band keyed on chunk `done/total`
|
||||
(message "Transcribing & translating").
|
||||
- Keep `onChunk` signature unchanged (no error return). Aborting on translate
|
||||
failure via the callback is intentionally avoided — see 1b.
|
||||
|
||||
After `Transcribe` returns:
|
||||
|
||||
- **Always store the transcript track** (unchanged, `service.go:447-466`,
|
||||
including the transcript `SubtitleReady` broadcast). This must run regardless
|
||||
of `translateErr`.
|
||||
- If `translateErr != nil`: `finishWithError(ctx, job, translateErr)`. The
|
||||
transcript is already persisted (cache preserved), and the live session
|
||||
already received `TranslationFailed`.
|
||||
- Else: assemble the translated track from `translatedAccum` (`sortCuesByStart`
|
||||
+ `StoreSubtitle`, `service.go:499-514`), `CompleteJob`, and
|
||||
`TranslationCompleted` (`service.go:515-522`).
|
||||
- Remove the whole-file translate-at-end (`service.go:479-497`) for the
|
||||
streaming path only.
|
||||
|
||||
Non-streaming chained jobs fall through to the existing whole-file path,
|
||||
untouched.
|
||||
|
||||
### 1b. Transcript-on-translation-failure (resolves review P1)
|
||||
|
||||
Today the full transcript is stored before translation starts
|
||||
(`service.go:447` then `:482`), so a translation failure still leaves a
|
||||
transcript/cache row and an already-broadcast `SubtitleReady`. The interleaved
|
||||
path preserves this exactly because:
|
||||
|
||||
- a per-chunk translate failure sets `translateErr` but lets the ASR pass run to
|
||||
completion (it does **not** return an error from `onChunk`), so `Transcribe`
|
||||
returns the full cue set normally;
|
||||
- the transcript is then stored on the same post-transcribe path as today,
|
||||
before the job is failed for the translated track.
|
||||
|
||||
An ASR failure or context cancellation still returns an error from `Transcribe`
|
||||
and yields no transcript — identical to today.
|
||||
|
||||
### 1c. Live progress semantics (resolves review P2)
|
||||
|
||||
`SubtitleTranslationCuesPayload.Done/Total` is documented as overall progress
|
||||
(`internal/playback/realtime.go:172-181`). For live `transcribe_translate`:
|
||||
|
||||
- `TranslationCues` carries **`done = chunks completed, total = total chunk
|
||||
count`** — the same chunk-granular convention plain `transcribe` already emits
|
||||
(`service.go:427-428`). Monotonic and stable across chunks.
|
||||
- `TranslationStarted.TotalCues = 0` (indeterminate), consistent with the
|
||||
transcribe path today (`service.go:412`), since the cue total is unknown before
|
||||
ASR completes.
|
||||
- Cues carry absolute Start/End, so the client places them regardless of
|
||||
arrival order; unpause is driven by cue arrival near the playhead, not by
|
||||
`Done/Total`. Confirm with client teams that `Done/Total` is treated as a
|
||||
fraction, not absolute cue indices (clients already receive chunk counts from
|
||||
the plain `transcribe` path, so this should hold).
|
||||
|
||||
### 1d. Small live chunks + clamp fix
|
||||
|
||||
- Add `TranscribeJobRequest.ChunkSeconds int` (0 = configured default). In
|
||||
`Transcribe`, use `req.ChunkSeconds` when `> 0`, else the atomic default
|
||||
(`transcriber.go:121-122`); validate/clamp identically to `SetExtraction`.
|
||||
- Fix the clamp in `SetExtraction` (`transcriber.go:99-105`): clamp
|
||||
out-of-range to the nearest bound, not to the 600 default.
|
||||
- Lower `minASRChunkSeconds` from 60 to 15 (`transcriber.go:26`); update the
|
||||
doc comment about request count / boundary word-clip tradeoff.
|
||||
- Config: add `subtitle_ai.live_asr_chunk_seconds` (default 30):
|
||||
- `internal/config/config.go`: new field beside `ASRChunkSeconds`
|
||||
(`config.go:274-277`).
|
||||
- `internal/config/db_loader.go`: load it next to `asr_chunk_seconds`
|
||||
(`db_loader.go:472-476`).
|
||||
- `internal/subtitles/ai/config.go`: add `LiveASRChunkSeconds int` to the
|
||||
service `Config` so the service can set `req.ChunkSeconds` for streaming jobs.
|
||||
- `internal/api/router.go`: plumb through `effectiveSubtitleAIConfig` and the
|
||||
`OnConfigChange` `UpdateConfig` (`router.go:1024-1055`).
|
||||
|
||||
### 1e. Phase 1 tests
|
||||
|
||||
- `service_test.go`: fake `Transcriber` emitting several chunks playhead-first
|
||||
via `onChunk`; fake `Translator` recording calls. For a streaming
|
||||
`transcribe_translate` job assert: one `Translate` per non-empty chunk; first
|
||||
`TranslationCues` streamed before the last chunk is transcribed; empty chunks
|
||||
skipped; final stored translated track = all translated cues sorted by start;
|
||||
`Done/Total` is chunk-granular and monotonic.
|
||||
- Failure case: a translate error on chunk N sets the failure but transcription
|
||||
completes, the transcript track **is** stored, `TranslationFailed` is emitted,
|
||||
and the job ends via `finishWithError`.
|
||||
- Non-streaming `transcribe_translate`: unchanged whole-file path; correct final
|
||||
track; no `TranslationCues`.
|
||||
- `transcriber_test.go`: `SetExtraction` clamp fix and `ChunkSeconds` override
|
||||
(incl. the new 15s floor).
|
||||
|
||||
---
|
||||
|
||||
## Phase 2 — playhead-first incremental extraction (true seconds latency)
|
||||
|
||||
Phase 1 still waits on full-file extraction before the first cue. Phase 2
|
||||
removes that by extracting and consuming chunks incrementally, seeked to the
|
||||
playhead.
|
||||
|
||||
### 2a. New incremental extractor in `internal/playback`
|
||||
|
||||
Add a streaming, seek-based extractor alongside `ExtractAudioChunks`:
|
||||
|
||||
```
|
||||
ExtractAudioChunksFrom(ctx, filePath, audioTrackIndex, dir, ffmpegPath,
|
||||
startSec float64, chunkSeconds int,
|
||||
onSegment func(AudioChunk) error) error
|
||||
```
|
||||
|
||||
- Runs one ffmpeg pass with `-ss <startSec> -i <file> ... -f segment
|
||||
-segment_list segments.csv` covering `startSec`→end.
|
||||
- Consumes segments **as they complete**: poll the segment-list CSV (ffmpeg
|
||||
appends `filename,start,end` when it closes each segment) on a short interval;
|
||||
for each new row, invoke `onSegment(AudioChunk{Path, Start: startSec + csvStart})`.
|
||||
Reconcile any trailing rows after `cmd.Wait()` so the final segment is not
|
||||
missed.
|
||||
- Honors `ctx` (CommandContext kills ffmpeg on cancel) and `onSegment` errors
|
||||
(stop the pass). Caller owns `dir`; segments are deleted by the consumer after
|
||||
ASR to cap disk at one extraction.
|
||||
- Preserves exact per-segment starts from the CSV, so timing does not accumulate
|
||||
drift within a pass (only the initial `-ss` seek is packet-accurate, sub-second,
|
||||
and is further corrected by the existing `ProbeAudioStartOffset`).
|
||||
|
||||
Rejected alternative — **per-chunk seek extraction** (`-ss s -t d` per chunk):
|
||||
simpler control flow but O(n) ffmpeg spawns and packet-aligned per-chunk seeks
|
||||
that can clip words at every seam. The single-pass streaming consumer keeps the
|
||||
existing segment-muxer boundary quality and far fewer process spawns.
|
||||
|
||||
### 2b. Transcriber incremental mode
|
||||
|
||||
- Add `TranscribeJobRequest.Incremental bool`, set by the service for streaming
|
||||
jobs.
|
||||
- When `Incremental`, `Transcribe` drives playhead-first in two streaming passes
|
||||
(reproducing today's `chunkOrderForPosition` order): pass A `startSec =
|
||||
playhead → end`, then pass B `0 → playhead`. Each completed segment is fed
|
||||
straight into the existing per-chunk ASR loop body (transcribe → `onChunk`),
|
||||
which Phase 1 already wires to translate+stream. The full cue set is still
|
||||
accumulated and returned for transcript storage.
|
||||
- When not `Incremental` (background, and plain transcribe unless we opt it in),
|
||||
keep the existing whole-file `ExtractAudioChunks` + `chunkOrderForPosition`
|
||||
path.
|
||||
|
||||
Note: timing composition (`-ss` reset PTS + segment CSV start + probe offset)
|
||||
must be verified with a fixture; see tests.
|
||||
|
||||
### 2c. Phase 2 tests
|
||||
|
||||
- `audio_extract` test (or transcriber test with an injected extractor) covering
|
||||
incremental emission order, exact-start mapping with a `startSec` offset, and
|
||||
ctx-cancel mid-pass.
|
||||
- Transcriber test asserting `Incremental` yields playhead-first chunk order
|
||||
across the two passes and returns the complete cue set for transcript storage.
|
||||
- Confirm first `onChunk` fires after ~one chunk of audio, not after full-file
|
||||
extraction (timing/ordering assertion via a fake extractor, not wall-clock).
|
||||
|
||||
---
|
||||
|
||||
## Multi-repo / client impact
|
||||
|
||||
None expected to the contract. Reuses the existing notifier protocol
|
||||
(`TranslationStarted` / `TranslationCues` / `TranslationCompleted` /
|
||||
`TranslationFailed`, `notifier.go`) under the same translated-track `trackKey`.
|
||||
Translated cues simply begin arriving far earlier, and `Done/Total` keeps the
|
||||
chunk-granular meaning plain `transcribe` already uses (1c). Worth a smoke test
|
||||
on one `silo-android` / `silo-apple` client to confirm the live track renders
|
||||
cues arriving before the transcript track completes, and that `Done/Total` is
|
||||
treated as a fraction.
|
||||
|
||||
## Risks & tradeoffs
|
||||
|
||||
- **Translation context at chunk boundaries (live only).** Per-chunk translation
|
||||
resets the batch context-neighbour window at each ~30s boundary — accepted
|
||||
quality tradeoff for live; background jobs are unchanged.
|
||||
- **More/smaller ASR requests for live jobs.** 30s chunks mean more requests;
|
||||
per-chunk timeout (`chunkSeconds * asrChunkTimeoutFactor`) and per-spawn
|
||||
overhead scale; the 15s floor bounds the worst case.
|
||||
- **Incremental extraction reliability (Phase 2).** Segment-list polling + the
|
||||
`-ss` seek path need fixtures for boundary timing, the final-segment race, and
|
||||
cancel/cleanup. Single ffmpeg pass per direction keeps cost low.
|
||||
- **Boundary word-clips.** Smaller live chunks add more fixed-length boundaries
|
||||
where a straddling word can clip — already a documented v1 limitation; the
|
||||
silence-aligned follow-up would address it.
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- **Pace to playback (issue item 3):** rolling window ahead of the playhead to
|
||||
bound work; separate change once interleave + incremental extraction land.
|
||||
- **Cross-chunk translation context:** feed the previous chunk's tail source
|
||||
cues as untranslated context into each chunk's translate call.
|
||||
- **Silence-aligned chunk boundaries:** reduce boundary word-clips that smaller
|
||||
live chunks make more frequent.
|
||||
|
||||
## v1 scope note
|
||||
|
||||
`docs/architecture/v1-scope.md` is **NOT LOCKED**; the issue carries
|
||||
`v1-proposed`. This is a **bug fix** to an already-shipped capability (no new
|
||||
user-facing capability, endpoint, or response field — only an internal config
|
||||
key and corrected streaming behavior), so it proceeds under the "bug fixes
|
||||
proceed normally" clause. Link the PR to the AI-subtitle capability item per the
|
||||
v1 PR requirements.
|
||||
|
||||
## Pre-push checklist
|
||||
|
||||
- `make lint`
|
||||
- `cd web && pnpm run lint && pnpm run format:check` (run only if web touched —
|
||||
none expected)
|
||||
- `make verify-local-paths`
|
||||
- Go tests for `internal/subtitles/ai` and `internal/playback` in a
|
||||
libvips-capable container (a bare-host `go test ./...` silently skips CGO
|
||||
packages).
|
||||
+188
@@ -0,0 +1,188 @@
|
||||
# Personal Collection Catalog Filter Standardization - Implementation Plan
|
||||
|
||||
- **Date:** 2026-06-23
|
||||
- **Spec:** `docs/superpowers/specs/2026-06-23-personal-collection-catalog-filter-standardization-design.md` (rev. 3, post-adversarial-review). The spec owns the *why* and the decisions; this plan owns the *how*, with concrete files and ordering.
|
||||
- **Note:** All paths are repository-relative. Commands assume the repository root is the cwd.
|
||||
|
||||
## Approach & PR Boundaries
|
||||
|
||||
Three concerns, landed in order. PR 1 is independently valuable and must merge first because PR 2 depends on its hardened predicate for correctness.
|
||||
|
||||
| PR | Concern | Depends on |
|
||||
|----|---------|-----------|
|
||||
| **PR 1** | Harden the catalog `watched` predicate (series/season rollup + row-type branching) and pin equivalence with item user-data. | — |
|
||||
| **PR 2** | Replace the `watch_filter` / `media_filter` enum vocabulary with a `display_query_definition` fragment end to end (migration → storage → API → execution → web). | PR 1 |
|
||||
| **PR 3** (deferred) | Wire display filters for smart/live collections via an executor-level `(source) AND (display)` conjunction. | PR 2 |
|
||||
|
||||
**Why PR 3 is deferred:** display-filter controls are currently exposed only for manual/exact and imported collections (`CollectionBuilder.tsx` renders them only when `collection_type === "manual"`; imported uses `ImportedCollectionEditor.tsx`). Smart collections never reach `FilterUserCollectionDisplayItems` today (`catalog_resolver.go` branches smart collections to `resolveQuerySource` before the filter call). So the OR-leak the adversarial review found is a guardrail for *future* wiring, not a v1 path. Build PR 2 so the same conjunction helper is reusable, but do not expose smart-collection display filters until PR 3.
|
||||
|
||||
---
|
||||
|
||||
## PR 1 — Catalog Watched Predicate Hardening
|
||||
|
||||
**Goal:** make the catalog `watched` rule produce the same played/read verdict as item user-data (`internal/api/handlers/user_state.go`) for every item kind, under both scoped and unscoped queries. This is the single-semantic foundation; everything else routes through it.
|
||||
|
||||
### Current state (verified)
|
||||
|
||||
- `internal/catalog/query_builder.go`:
|
||||
- `buildWatchedClause` (≈900–912) wraps `userStateCompletionClause` and negates for `value:false`.
|
||||
- `userStateCompletionClause` (≈1119–1171) checks completion against **the row's own `content_id`** (`uwp.media_item_id = mi.content_id` / `uwh.media_item_id = mi.content_id`). No episode rollup → a series row is effectively never watched.
|
||||
- `ebookUserStateCompletionClause` (≈1173–1193) is selected **only when `qb.mediaScope == "ebook"`** → an ebook under an unscoped query is misclassified by the video clause.
|
||||
- `internal/api/handlers/user_state.go` `allEpisodesCompleted` (≈171–185) defines the canonical rollup over the episode repository's set.
|
||||
- Episode set (the parity target): `internal/catalog/episode_repo.go` `ListBySeriesIDs` / `ListBySeasonIDs` select from `episodes` filtered by `series_id` / `season_id` **AND** `episodeAvailabilityPredicate` = `EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = episodes.content_id)`. No specials/season-0 exclusion, no media-file requirement. Links: `episodes.series_id`, `episodes.season_id`.
|
||||
- The base relation always exposes `mi.type` (`media_items mi`, or the episode derived table that hard-codes `'episode'::text AS type`), so a `CASE` on row type is feasible.
|
||||
|
||||
### Changes
|
||||
|
||||
1. **Extract a reusable completion predicate.** In `internal/catalog/query_builder.go`, refactor `userStateCompletionClause` into a helper that, given a row alias and the linking column, emits the leaf completion EXISTS pair (progress + history, with hidden-history guard). Keep `ebookUserStateCompletionClause` as the ebook leaf. This is the shared building block for the branches below.
|
||||
|
||||
2. **Add series/season rollup branches with repository-parity episode sets.** Emit, for a series row, "≥1 child episode exists AND no child episode is incomplete," expressed so the child-episode set is **identical** to `episodeAvailabilityPredicate`:
|
||||
```sql
|
||||
EXISTS (SELECT 1 FROM episodes e
|
||||
WHERE e.series_id = mi.content_id
|
||||
AND EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = e.content_id))
|
||||
AND NOT EXISTS (
|
||||
SELECT 1 FROM episodes e
|
||||
WHERE e.series_id = mi.content_id
|
||||
AND EXISTS (SELECT 1 FROM episode_libraries el WHERE el.episode_id = e.content_id)
|
||||
AND NOT (<leaf completion predicate for e.content_id>))
|
||||
```
|
||||
Season rollup is the same with `e.season_id = mi.content_id`. The leaf predicate inside the `NOT EXISTS` reuses the helper from step 1 keyed on `e.content_id`. **The `episode_libraries` EXISTS must match `episodeAvailabilityPredicate` verbatim** — if `episode_repo.go` ever changes that predicate, this SQL changes with it. Add a code comment cross-referencing `episodeAvailabilityPredicate` so the coupling is discoverable.
|
||||
|
||||
3. **Branch by row type for unscoped queries.** When `qb.mediaScope` is empty/`video` (rows can be mixed), select the completion source from `mi.type` rather than from `mediaScope`:
|
||||
- `movie` / `episode` / video leaf → leaf predicate,
|
||||
- `series` → series rollup, `season` → season rollup,
|
||||
- `ebook` → ebook leaf,
|
||||
- `manga` → deferred; treat as never-completed for now (documented), do not invent a collection-only interpretation.
|
||||
When `mediaScope` is a single concrete kind (e.g. `series`, `ebook`), the optimizer-friendly path may emit just that branch. Implement as a `CASE mi.type WHEN ... END` or chained `(mi.type = 'x' AND <clause>) OR ...`; verify argIdx accounting (each leaf/ebook clause increments `qb.argIdx` by 2 — the rollups reuse the leaf, so count the placeholders carefully and add a builder test that the produced arg count matches bound args).
|
||||
|
||||
4. **`last_watched` stays separate.** Do not route `last_watched` through the new predicate; it is a `user_last_watched` CTE timestamp aggregate. Leave as-is; add a one-line comment noting the intentional divergence.
|
||||
|
||||
### Tests (PR 1)
|
||||
|
||||
- `internal/catalog/query_builder_test.go`: assert the `watched` clause contains the series/season rollup EXISTS only when the scope/row-type can include series/seasons; assert arg count == bound args for each branch; assert the `episode_libraries` predicate text is present in the rollup.
|
||||
- **Equivalence test (highest value):** seed a Postgres fixture with a series whose episodes are variously completed/not, a season, a movie, and an ebook for one profile; assert the catalog `watched` rule's matched set equals exactly the items whose `user_state.go` `Played` flag is true. Because `user_state.go` lives in `internal/api/handlers` (CGO/libvips), run this in the libvips-capable container; the test can drive both the catalog executor and the item-user-state builder against the same pool and diff the verdicts. If a same-package home is awkward, place the comparison test in `internal/api/handlers` where both sides are reachable.
|
||||
|
||||
### Release note (PR 1)
|
||||
|
||||
Catalog `watched` filtering for series/seasons changes from "never matches" to "matches when all available episodes are completed." Call this out in the PR body and release notes per the spec's semantics-change risk.
|
||||
|
||||
---
|
||||
|
||||
## PR 2 — Replace the Enum Vocabulary with `display_query_definition`
|
||||
|
||||
One concern: swap the persisted representation and every surface that reads/writes it. Clean replacement — no aliases, no backfill (the enums are branch-only and unreleased; see spec Current Branch State). Land the steps below together so no layer reads a column another layer has dropped.
|
||||
|
||||
### Step 2.1 — Migration reshape
|
||||
|
||||
Edit `migrations/sql/20260623184858_user_collection_display_filters.sql` in place so the canonical schema has no legacy columns:
|
||||
|
||||
- **Up:** `ALTER TABLE public.user_personal_collections ADD COLUMN IF NOT EXISTS display_query_definition jsonb;` (nullable; `NULL`/absent = no display filter). Remove the `watch_filter` / `media_filter` `ADD COLUMN` and their CHECK constraints.
|
||||
- **Down:** `ALTER TABLE ... DROP COLUMN IF EXISTS display_query_definition;`
|
||||
|
||||
**Operational gotcha:** Goose keys applied migrations by version, so editing an already-applied migration will **not** re-run it. Any dev DB that already applied the original `20260623184858` (which added `watch_filter`/`media_filter`) must be rolled back to before it and re-applied: `make migrate-down` to the prior version, then `make migrate-up`. If a shared dev DB makes in-place editing risky, the alternative is a *new* timestamped migration (`make migrate-create NAME=replace_collection_display_filters`) that drops the two columns and adds `display_query_definition`; the spec prefers in-place reshape since nothing is merged, but call the choice out in the PR.
|
||||
|
||||
### Step 2.2 — Backend storage & types
|
||||
|
||||
- `internal/userstore/types.go`: in `Collection` (≈185–213), `CreateCollectionInput` (≈238–255), `UpdateCollectionInput` (≈257–278), remove `WatchFilter` / `MediaFilter`; add `DisplayQueryDefinition string` (and `*string` on the update input), mirroring the existing `QueryDefinition` fields.
|
||||
- `internal/userstore/pgstore/collections.go`:
|
||||
- `collectionSelectColumns` (≈15–18): drop `watch_filter, media_filter`; add `display_query_definition`.
|
||||
- `scanCollection` (≈20–42) and the `ListCollections` scan (≈192–196): drop the two scan targets; add `&c.DisplayQueryDefinition`.
|
||||
- `CreateCollection` INSERT (≈79–94): drop the two columns/args; add `display_query_definition` + arg.
|
||||
- `UpdateCollection` (≈270–283): replace the two `add("watch_filter"…)` / `add("media_filter"…)` blocks with a single `display_query_definition` block driven by `input.DisplayQueryDefinition != nil`.
|
||||
- The normalization at create-time (≈59–68) moves from enum normalization to **fragment normalization** (Step 2.5).
|
||||
- `internal/userstore/collection_filters.go`: repurpose into the fragment normalizer's home, or delete the enum constants/`Normalize*` funcs if nothing else uses them after the swap. Keep the file's tests meaningful — convert `collection_filters_test.go` to cover the new normalizer (Step 2.5) rather than the deleted enums.
|
||||
|
||||
### Step 2.3 — API surface
|
||||
|
||||
- `internal/api/handlers/collections.go`:
|
||||
- `createCollectionRequest` (≈39–50), `updateCollectionRequest` (≈52–67), `collectionResponse` (≈73–101): drop `watch_filter` / `media_filter`; add `display_query_definition json.RawMessage` following the existing `query_definition` handling (response assembly mirrors `defaultJSON([]byte(c.QueryDefinition))` at ≈843).
|
||||
- `HandleCreateCollection` (≈270–279) / `HandleUpdateCollection` (≈341–356): replace enum validation with fragment normalization/validation (Step 2.5); reject invalid fragments with `400`.
|
||||
- `collectionCapabilitiesResponse` (≈108–111) + `HandleCapabilities` (≈224–229): replace `watch_filters`/`media_filters` value lists with capability metadata describing the supported display-query fields (`type`, `watched`) and the UI presets (`all/watched/unwatched`, `all/movies/series`). Keep it additive and feature-detection-oriented per the v1 API rules.
|
||||
- `internal/api/handlers/user_collection_imports.go`: `userImportSharedFields` (≈67–77) and `createImportedCollection` validation (≈214–223) — same swap to `display_query_definition`.
|
||||
|
||||
### Step 2.4 — Exact-collection execution through the catalog executor
|
||||
|
||||
Replace the custom Go filter pass and the metadata-only in-Go matcher with one SQL path.
|
||||
|
||||
- Delete `internal/catalog/user_collection_display_filters.go` (`FilterUserCollectionDisplayItems` and helpers) and its test, plus the duplicated `userCollectionAllEpisodesCompleted`.
|
||||
- Call sites to rewrite:
|
||||
- `internal/catalog/catalog_resolver.go` `resolveUserCollectionSource` (≈479–526, filter call at ≈517).
|
||||
- `internal/sections/fetcher.go` `fetchUserCollection` (filter call at ≈1371).
|
||||
- New behavior for exact collections (per spec Execution Model):
|
||||
1. Load member IDs in source order (`user_personal_collection_items`, `ORDER BY position`) — unchanged.
|
||||
2. Parse `collection.DisplayQueryDefinition` into a `QueryDefinition`, **force `Limit = nil`** and drop any fragment `sort`/`library_ids` defensively before execution.
|
||||
3. Run the executor with `AccessFilter.AllowedContentIDs = memberIDs` (the pattern `resolveCandidateItemsWithQuery` already uses at `catalog_resolver.go` ≈606–607). The executor's `conditions` slice AND-joins `AllowedContentIDs` with the display fragment, so membership ∧ filter is automatic — no OR-leak risk here.
|
||||
4. For source-ordered requests: request the **full** match set (no limit), collect matching IDs into a set, reapply original order, then apply request offset/limit. Pagination happens after reordering, never inside the executor.
|
||||
5. Stop using `filterCatalogItems` (`catalog_resolver.go` ≈1772) for these collections — replace the `UseSourceOrder` branch in `resolveExactOrderedMediaItems` (≈571–586) with the match-set-then-reorder path. **Do not delete `filterCatalogItems`:** it has a second caller at `catalog_resolver.go:244` (the query-source path), so it stays; only the exact-collection call at ≈575 changes.
|
||||
- Empty display fragment ⇒ skip the executor filter entirely and return members in source order (preserves today's "no filter" performance for unfiltered collections).
|
||||
|
||||
### Step 2.5 — Fragment normalization (shared backend helper)
|
||||
|
||||
A `display_query_definition` is a **filter-only** `QueryDefinition` fragment. Add one normalizer (e.g. in `internal/catalog` next to `QueryDefinition`, or `internal/userstore`) used by every write path (create/update/import):
|
||||
|
||||
- Accept only `match` + `groups`, with rules limited to the allowed display vocabulary (v1: `type`, `watched`).
|
||||
- Reject (preferred, on write) or strip `limit`, `sort`, `library_ids`, `media_scope`.
|
||||
- Reject unknown fields/ops so clients learn the contract.
|
||||
- Return a canonical JSON string for storage. Used by `pgstore` create/update and the API handlers.
|
||||
|
||||
### Step 2.6 — Web
|
||||
|
||||
The frontend already has the machinery: `QueryDefinition` types, `createEmptyQueryDefinition` / `normalizeQueryDefinition` (`web/src/api/types.ts` ≈1318–1361), and `CollectionGuidedRulesEditor.tsx` which already converts `watched` rules ↔ a `watchStatus` field (`queryDefinitionToGuidedState` ≈93–254, `guidedStateToQueryDefinition` ≈257–374). Reuse it; do not add a parallel state model.
|
||||
|
||||
- `web/src/lib/collectionDisplayFilters.ts`: replace the enum option lists / `normalize*` helpers with converters **preset ⇄ `QueryDefinition` fragment**:
|
||||
- `watched`/`unwatched`/`all` ⇄ a `{field:"watched",op:"is",value:true|false}` rule (or none),
|
||||
- `movie`/`series`/`all` ⇄ a `{field:"type",op:"is",value:"movie"|"series"}` rule (or none),
|
||||
- a builder that assembles the two presets into a single filter-only fragment, and a reader that derives the two preset values back from a fragment (for editing).
|
||||
- `web/src/api/types.ts`: drop `UserCollectionWatchFilter` / `UserCollectionMediaFilter` and the `watch_filter` / `media_filter` fields on `Collection` (≈1270–1271), `CreateCollectionRequest` (≈1403–1404), `UpdateCollectionRequest` (≈1420–1421); add `display_query_definition?: QueryDefinition`.
|
||||
- Editors — swap the two `<Select>` controls' wiring to read/write through the helper, keeping the same visible labels:
|
||||
- `web/src/components/collections/CollectionBuilder.tsx` (value shape ≈45–57, UI ≈259–313, submit ≈88–101),
|
||||
- `web/src/pages/ImportedCollectionEditor.tsx` (state ≈118–125, UI ≈277–354, save ≈184–214),
|
||||
- `web/src/components/CollectionTemplateGallery/UserCollectionTemplateConfigForm.tsx` (state ≈99–100, submit ≈119–167).
|
||||
- `web/src/hooks/queries/collections.ts` (`useCreateCollection` ≈85–101, `useUpdateCollection` ≈103–127): payloads now carry `display_query_definition` instead of the enums (the generic `buildUserCollectionPayload` needs no change beyond the type swap).
|
||||
|
||||
### Tests (PR 2)
|
||||
|
||||
- `internal/api/handlers/collections_test.go`: create/update/get round-trips `display_query_definition`; `watch_filter`/`media_filter` are absent from request and response; an invalid fragment (e.g. containing `limit` or an unknown field) returns `400`.
|
||||
- Resolver/executor tests:
|
||||
- exact collection routes through the catalog `watched` rule and preserves source order across pages;
|
||||
- an exact collection larger than a page returns the full match set in source order regardless of fragment contents (guards the limit-truncation finding);
|
||||
- a mixed-media exact collection (movie + ebook) with `watched`/`unwatched` classifies the ebook via `ebook_reader_progress`, not video history (relies on PR 1 row-type branching).
|
||||
- Fragment-normalizer unit tests (replacing `collection_filters_test.go`): strips/rejects `limit`/`sort`/`library_ids`/`media_scope`; round-trips `type` + `watched` rules.
|
||||
- Frontend: helper test for preset ⇄ fragment conversion (both directions); one editor test asserting the submit payload carries the canonical `display_query_definition`.
|
||||
|
||||
---
|
||||
|
||||
## PR 3 — Smart/Live Display Filters (deferred)
|
||||
|
||||
Only when smart/live collections expose display-filter controls. Implements the spec's executor-level conjunction so `(source) AND (display)` holds regardless of the source's top-level `match`.
|
||||
|
||||
- Seam: `internal/catalog/query_executor.go`, after the `AllowedContentIDs` handling (≈322) and before `ApplySectionAccessFilter`. Build the display fragment with a second `NewQueryBuilder("mi").WithArgIdx(argIdx)`, `rebindSQLPlaceholders` its output to the running `argIdx`, append the rebound clause to `conditions`, and advance `argIdx`/`args`. Because `conditions` is AND-joined (≈353), this yields `(sourceWhere) AND (displayWhere)` with no group-list manipulation and no OR-leak.
|
||||
- Do **not** append the display group into `def.Groups` — that is the OR-leak the adversarial review found (`QueryBuilder.Build` joins top-level groups with the def's single `match`, ≈84–111).
|
||||
- Tests: a `match: any` source plus a display filter never returns rows the source alone excludes; a contradictory `watched` overlay on a `match: any` source returns zero rows.
|
||||
|
||||
---
|
||||
|
||||
## Verification (every PR, pre-push)
|
||||
|
||||
Per `CLAUDE.md`:
|
||||
|
||||
- `make lint`
|
||||
- `cd web && pnpm run lint && pnpm run format:check`
|
||||
- `make verify-local-paths`
|
||||
- Go tests in a **libvips-capable container** (a bare-host `go test ./...` silently skips CGO packages including `internal/api/handlers`, where the equivalence and DTO tests live). Prefix `GOWORK=off` for builds/tests in this repo.
|
||||
- Targeted: `go test ./internal/catalog/... ./internal/api/handlers/... ./internal/userstore/...` and the relevant `web` helper/editor tests.
|
||||
|
||||
## Sequencing summary
|
||||
|
||||
1. PR 1 (predicate + equivalence test) → merge.
|
||||
2. PR 2 (migration → types/storage → API/imports → fragment normalizer → exact execution → web), landed together → merge.
|
||||
3. PR 3 only if/when smart-collection display filters are wired.
|
||||
|
||||
Link each PR to the capability sub-issue (`Part of #NNN`) per the v1 process; keep one concern per PR.
|
||||
|
||||
## Open items to confirm before/while implementing
|
||||
|
||||
- Confirm no `silo-android` / `silo-apple` build already consumes the branch's `watch_filter` / `media_filter` fields (spec assumes unreleased; if pinned, fall back to the contingency backfill path in the spec's Migration Strategy).
|
||||
- Confirm `episodeAvailabilityPredicate` is the intended "child episode" definition to match (Open Question #4 in the spec) — it currently includes specials/season-0 if they have library links. Match it exactly; change item user-data and the catalog rule together if it should differ.
|
||||
- Decide the fragment normalizer's package home (`internal/catalog` vs `internal/userstore`) based on import direction — it needs the `QueryDefinition` type without creating an import cycle.
|
||||
@@ -0,0 +1,327 @@
|
||||
# Hybrid Semantic Search Hardening Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. Commands assume the repository root is the cwd.
|
||||
|
||||
**Goal:** Make Meilisearch hybrid semantic search safe to enable broadly by gating it on *trustworthy* per-type vector coverage, validating embedder capability, and exposing per-query/admin diagnostics — without regressing the hot search path.
|
||||
|
||||
**Architecture:** Semantic search stays gated behind the existing `SemanticEnabled && wordCount >= 3` rule, plus a new **coverage gate** that reads an in-memory, periodically-refreshed per-type coverage snapshot (never a per-request DB query). Coverage = `current-model embeddings ÷ embed-eligible items`, computed entirely from Postgres and latched with hysteresis. Existing provider/fallback fields are plumbed through the resolver into an additive `search_diagnostics` response object. A rate-limited capability probe validates the Meili embedder without downing keyword search.
|
||||
|
||||
**Tech Stack:** Go (pgx/pgxpool), Meilisearch hybrid search (`userProvided` embedder), React/TypeScript admin UI, Goose migrations.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- **API additive-only within `/api/v1`** — never rename/remove a response field, change a type, or repurpose a status code. New fields are `omitempty` additions. (CLAUDE.md)
|
||||
- **Performance first, reliability first, predictable under load** — the search request path must not gain a synchronous DB count, lock wait, or remote probe. Fail **safe** (keyword-only), never panic. (CLAUDE.md Core Priorities)
|
||||
- **One additive index migration** is in scope (revised — see Assumptions): a btree on `media_item_embeddings(model)` to support the periodic coverage query. No schema/column changes.
|
||||
- **Canonical embedding dimensions** = `embeddingvectors.CanonicalDimensions` (= 3072; re-exported as `recommendations.CanonicalEmbeddingDimensions`). Never hardcode the number.
|
||||
- **Single source of truth for embed-eligibility** — the predicate `(status='matched' OR type IN ('audiobook','ebook'))` must not be duplicated. It moves to the neutral `internal/embeddingvectors` package (imported by both `catalog` and `recommendations`; `recommendations` already imports `catalog`, so `catalog` cannot import `recommendations`).
|
||||
- **Single phase** — measurement *and* enforcement ship together. The diagnostics shipped here are what let us retune thresholds later without code changes-in-anger.
|
||||
- Build/test in worktrees with `GOWORK=off` and a stubbed `web/dist` (see worktree-build-quirks memory).
|
||||
|
||||
---
|
||||
|
||||
## Corrected Baseline (what actually exists today — verified)
|
||||
|
||||
Work from this, not from intuition. Each point was confirmed against the tree.
|
||||
|
||||
1. **The only semantic skip rule is `SemanticEnabled && len(Fields(normalizedQuery)) >= 3`** (`internal/catalog/search_meilisearch_provider.go:329`). There is **no** `SkipTotal`/preview skip, and `TestMeilisearchSearchRequestBuildsHybridForApproximateInteractiveSearch` (`search_provider_test.go:213`) asserts hybrid **IS** used with `SkipTotal: true`. → **Do not add a `SkipTotal` skip.** Layer the coverage gate onto the existing rule; remove nothing.
|
||||
2. **`CatalogSearchResult` already carries `Provider` and `FallbackReason`** (`search_provider.go:64-71`). Diagnostics work is *plumbing these through* `CatalogResult` (which drops them, `catalog_resolver.go:24-30,291-296`) — not inventing provider fields.
|
||||
3. **The provider already self-downgrades hybrid→keyword on a Meili error** (`search_meilisearch_provider.go:221-228`). Diagnostics must report the *post-downgrade* reality.
|
||||
4. **`countCatalogSearchVectorDocuments` ignores `model`** (`search_indexer.go:757`), counts the *indexed* population (no eligibility filter), and its companion denominator (`document_count`) comes from **Meili index stats** (`search_service.go:129`), not Postgres. It has **four callers** — `search_service.go:139`, `search_indexer.go:133`, `:203`, `:310` — all must be updated when its signature changes.
|
||||
5. **Indexed population ≠ embed-eligible population.** `LoadDocumentsAfter` (`search_indexer.go:534`) indexes every type-matched, non-manga row regardless of `status`; embeddings are only written for `(status='matched' OR audiobook OR ebook)` (`recommendations/repo.go:154-160`, `recommendationItemEligibilityWhereClause`). A denominator of "indexable items" is therefore structurally < 1.0 on any library with unmatched video → **the gate would never open.** Coverage must use the **embed-eligible** population for both numerator and denominator.
|
||||
6. **Status fields that already exist:** `document_count`, `vector_document_count` (`search_provider.go:347-348`). The earlier draft's `index_document_count` does **not** exist — keep real names; add new fields alongside.
|
||||
7. **`UpsertEmbedding` already enqueues a Meili index event after every write** (`recommendations/repo.go:186`) — verified correct, **no change**.
|
||||
8. **Catalog search settings are restart-bound** (`config/restart_keys.go:75-85`, *"Catalog search provider construction is intentionally startup-bound in v1"*). The service is built **once** in `NewRouter` and never live-rebuilt. → the coverage refresher is a single process-lifetime goroutine; no live-rebuild leak exists in v1.
|
||||
9. **`catalogSearchService` is a local var in `NewRouter`** (`router.go:437`), wired into handlers but **never returned** to `main.go`. The long-lived `deps.AppContext` (`router.go:88`) is the refresher's context.
|
||||
10. **Legacy `/search` plain-`q` re-route is descoped** — see [Descoped](#descoped--not-in-this-pass).
|
||||
|
||||
---
|
||||
|
||||
## Design Decisions
|
||||
|
||||
### D1 — Coverage = current-model embeddings ÷ embed-eligible items (single Postgres source, per type)
|
||||
|
||||
Let `ELIG(alias)` = `embeddingvectors.ItemEligibilityWhereClause(alias)` → `(alias.status='matched' OR alias.type IN ('audiobook','ebook'))` (moved here from `recommendations`; `recommendations.recommendationItemEligibilityWhereClause` is refactored to call it — no behavior change, no duplication).
|
||||
|
||||
- **Denominator** (embed-eligible items, per type):
|
||||
```sql
|
||||
SELECT mi.type, COUNT(*) AS eligible
|
||||
FROM media_items mi
|
||||
WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id)
|
||||
AND ($1::text[] IS NULL OR mi.type = ANY($1))
|
||||
AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi")
|
||||
GROUP BY mi.type
|
||||
```
|
||||
- **Numerator** (eligible items with a *current-model* embedding, per type):
|
||||
```sql
|
||||
SELECT mi.type, COUNT(*) AS vectorized
|
||||
FROM media_item_embeddings e
|
||||
JOIN media_items mi ON mi.content_id = e.media_item_id
|
||||
WHERE NOT EXISTS (SELECT 1 FROM manga_chapters mc WHERE mc.chapter_content_id = mi.content_id)
|
||||
AND ($1::text[] IS NULL OR mi.type = ANY($1))
|
||||
AND (mi.status = 'matched' OR mi.type IN ('audiobook','ebook')) -- ELIG("mi")
|
||||
AND ($2 = '' OR e.model = $2)
|
||||
GROUP BY mi.type
|
||||
```
|
||||
|
||||
Applying `ELIG` to **both** guarantees numerator ⊆ denominator → ratio ∈ `[0,1]` (an embedding left behind on an item that became unmatched can't push a type over 1.0). Unmatched items are excluded from both, so they never drag coverage. As backfill completes, ratio → 1.0.
|
||||
|
||||
**Active model (`$2`)** is read from the embedding lock (`recommendations.EmbeddingLock.Model`, `embedding_lock.go:16`). No lock ⇒ `$2 = ""` and the snapshot is marked not-ready (never gate semantic on for an unestablished embedding space).
|
||||
|
||||
> **Why model-filtering is load-bearing:** right after a model change, old vectors persist but are mismatched to the new query embedder. A model-blind count reads ~100% at the most dangerous moment. Filtering by the locked model collapses coverage to ~0 until re-embedding catches up.
|
||||
|
||||
**Index (migration, in scope):** `media_item_embeddings` has only an HNSW index on `embedding` (`migrations/sql/001_schema.sql:1857`, `014_...:25`) — nothing on `model`. Add `CREATE INDEX CONCURRENTLY idx_media_item_embeddings_model ON public.media_item_embeddings (model)` so the numerator filter is index-supported. (Goose: timestamped file via `make migrate-create`, with `-- +goose NO TRANSACTION` because `CONCURRENTLY` cannot run in a txn.)
|
||||
|
||||
### D2 — Coverage lives in an in-memory snapshot, refreshed off the request path
|
||||
|
||||
A `semanticCoverageTracker` (new, `internal/catalog/semantic_coverage.go`) holds an `atomic.Pointer[semanticCoverageSnapshot]`.
|
||||
|
||||
- **Refresh cadence:** a single background goroutine every `semanticCoverageRefreshInterval` (**2 minutes** — long enough that two `GROUP BY` aggregates are negligible steady-state load even at 1M rows; short enough to bound the post-model-change danger window, which also fails safe via the model-collapse rule below). The interval is a named const; **there is no after-sync trigger** (the indexer and service share no handle — see Descoped rationale; the interval alone is sufficient and avoids a cross-component wiring + race).
|
||||
- **Single-flight & immutability:** `Refresh` holds a `sync.Mutex` across read-prev → query → compute → publish (background path; contention irrelevant). Each refresh builds a **fresh** `PerType` map and publishes a new immutable snapshot pointer; the published snapshot is **never mutated** after publish (avoids a `map` data race against concurrent `CoverageReady` reads). A `-race` test fires refreshes concurrently with reads.
|
||||
- **Fail-safe reads:** `CoverageReady` reads only the atomic pointer — **no DB, no lock, no remote call.** If the snapshot is `nil` (boot, before first refresh) it returns `(false, "coverage not yet computed")` — **never** dereferences a nil map (that would be a hot-path panic).
|
||||
- **Error retention:** on a refresh query error, retain the last-good snapshot (transient DB blips must not flip semantic off); log; do not publish a zeroed snapshot. Admin status flags `UpdatedAt` staleness.
|
||||
- **Model-change collapse:** the snapshot records the `Model` it was computed for. When `Refresh` observes the active model differs from the published snapshot's model, it **immediately publishes an all-not-ready snapshot** (safe) and recomputes real coverage under the new model on the same pass. This bounds the mismatched-vector window to ≤ one interval and fails safe within it.
|
||||
|
||||
### D3 — Per-type readiness with hysteresis; scope = AND of its types; empty scope = all types
|
||||
|
||||
Each type's `ready` flag is **latched**: flips **true** at `ratio >= semanticCoverageEnableRatio` (0.90), flips **false** only at `ratio < semanticCoverageDisableRatio` (0.80). The snapshot carries prior latched state so each refresh applies hysteresis — preventing flap (identical queries returning differently-ranked results).
|
||||
|
||||
`CoverageReady(itemTypes)`:
|
||||
- **Non-empty scope:** ready iff **every** requested type is latched-ready (`min` semantics — a weak subtype blocks the mixed scope). The reason names the first failing type.
|
||||
- **Empty/nil scope** (the common "search everything" case — `MediaScopeItemTypes("") → nil`, so `ItemTypes == nil` reaches the gate): ready iff **every type in the snapshot** is latched-ready. This deliberately mirrors the empty-config semantics and refuses to let an unscoped search bypass a weak type. Reason names the first failing type, or `"coverage not yet computed"` if the snapshot is nil/empty.
|
||||
|
||||
### D4 — Diagnostics: existing fields, plumbed, reported post-downgrade, scoped to the provider path
|
||||
|
||||
- Add `Mode string` (`"keyword"`/`"hybrid"`) and `SemanticUsed bool` to `CatalogSearchResult`; the Meili provider sets them to reflect the **final** request after the internal hybrid→keyword downgrade.
|
||||
- Add `Provider`, `Mode`, `SemanticUsed`, `FallbackReason` to `CatalogResult`; copy them in `resolveDirectSearchSource`.
|
||||
- Add an `omitempty` `search_diagnostics` object to `catalogResponse`.
|
||||
- **Scope:** diagnostics are produced **only** on the direct provider-backed search path (`useDirectSearchPath` = relevance sort, desc, no advanced rules — `catalog_resolver.go:1416-1427`). They are **omitted** for: browse, non-relevance-sorted `q=` searches (which never hit the provider), and `group=work` responses (the grouped wrapper builds a fresh `CatalogResult` at `catalog.go:287-293` and is already approximate — `TotalExact=false`). Each exclusion gets an explicit handler test.
|
||||
|
||||
### D5 — Capability validation: narrow, rate-limited, non-fatal, admin-path only
|
||||
|
||||
Embedder settings are written by Silo itself, so validation guards only **external drift** (Meili volume reset, manual edits). Add `GetSettings` to the Meili client; verify the configured embedder exists, `source == "userProvided"`, and `dimensions == embeddingvectors.CanonicalDimensions`. The hybrid probe:
|
||||
- runs **only on the admin status path** (`HandleGetCatalogSearchStatus`), **not** on `CheckConnection`/health (which only does `GET /health`), and the admin query has no `refetchInterval` (`settings.ts`, `staleTime: 15_000`);
|
||||
- runs **only when `state.ActiveIndexUID != ""`**, and is **cached for `semanticCapabilityProbeTTL` (5m)** → ≤ 1 probe / 5m regardless of refresh rate;
|
||||
- uses a **non-degenerate unit sample vector** (`v[0]=1.0`, rest 0, length `CanonicalDimensions`) — an all-zero vector has zero norm and can spuriously fail cosine hybrid; targets `ActiveIndexUID` with `Hybrid.Embedder = config.Embedder`, `limit:1`.
|
||||
|
||||
Failures surface in admin status and force keyword-only; they **never** flip provider `Healthy` or trip the circuit.
|
||||
|
||||
### D6 — Backfill split: cheap discovery vs. bounded text-staleness, with fairness
|
||||
|
||||
The expensive part of `ListEmbeddingTextCandidates` is **building `current_text` for every eligible row** via 5 `LATERAL` joins to `item_people` (`recommendations/repo.go:544-589`) — *not* the `IS DISTINCT FROM` comparison.
|
||||
|
||||
- **Cheap discovery** (`ListMissingOrModelStaleEmbeddingIDs`): `LEFT JOIN media_item_embeddings`, filter `ELIG("mi") AND (e.media_item_id IS NULL OR e.model = '' OR e.model <> $currentModel)`, `ORDER BY mi.content_id`, `LIMIT`. **No `LATERAL`, no `current_text`.** The SQL is built by a package-private `buildMissingOrModelStaleEmbeddingIDsSQL() string` helper so a unit test can assert it contains neither `item_people` nor `LATERAL` (mirrors the existing `embeddingEligibilityWhereClause()` string-helper test pattern). Returns IDs; `current_text` is then built **only for that bounded batch** via `BuildEmbeddingTextForIDs(ids)`.
|
||||
- **Text-staleness** (existing `ListEmbeddingTextCandidates`): keep, but run as a **bounded, unconditional quota per cycle** (`embeddingTextStaleQuotaPerRun`) so canonical-text-stale items progress even when the cheap backlog never empties (fairness against starvation).
|
||||
- **Cursors:** the cheap pass advances `afterID` from the **cheap query's** last id (it is the ordered cursor), independent of `BuildEmbeddingTextForIDs` row order (`ANY($1)` does not preserve slice order, and a dropped id must not desync the cursor). The text-stale pass keeps its **own** `afterID`. Both monotonic.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility | Change |
|
||||
| --- | --- | --- |
|
||||
| `migrations/sql/<ts>_media_item_embeddings_model_index.sql` | `(model)` btree, `CONCURRENTLY` / `NO TRANSACTION` | Create |
|
||||
| `internal/embeddingvectors/eligibility.go` | `ItemEligibilityWhereClause(alias)` — single source of truth | Create |
|
||||
| `internal/catalog/semantic_coverage.go` | tracker, snapshot, querier seam, per-type SQL, hysteresis, single-flight refresher | Create |
|
||||
| `internal/catalog/search_provider.go` | gate + model-provider interfaces; `Mode`/`SemanticUsed`; semantic status structs | Modify |
|
||||
| `internal/catalog/search_meilisearch_provider.go` | consult gate; set `Mode`/`SemanticUsed` (incl. downgrade); capability probe (unit vector) | Modify |
|
||||
| `internal/catalog/search_meilisearch_client.go` | `GetSettings` + settings DTOs | Modify |
|
||||
| `internal/catalog/search_service.go` | build/own tracker; `StartCoverageRefresh`; comma-ok model-provider; `Status()` reads snapshot | Modify |
|
||||
| `internal/catalog/search_indexer.go` | `ELIG`+`model` numerator & eligible denominator; update **all 4** `countCatalogSearchVectorDocuments` callers | Modify |
|
||||
| `internal/catalog/catalog_resolver.go` | carry diagnostics through `CatalogResult` (direct path only) | Modify |
|
||||
| `internal/recommendations/repo.go` | delegate eligibility to `embeddingvectors`; `ListMissingOrModelStaleEmbeddingIDs` (+SQL helper); `BuildEmbeddingTextForIDs` | Modify |
|
||||
| `internal/recommendations/similar.go` | `EmbedAll`: cheap drain + unconditional text-stale quota; dual cursors | Modify |
|
||||
| `internal/recommendations/engine.go` | `ActiveEmbeddingModel(ctx)` from the lock | Modify |
|
||||
| `internal/api/handlers/catalog.go` | `searchDiagnostics` DTO; populate on direct path; omit for grouped | Modify |
|
||||
| `internal/api/router.go` | start `service.StartCoverageRefresh(deps.AppContext)` inside `NewRouter` | Modify |
|
||||
| `web/src/hooks/queries/admin/settings.ts`, `web/src/pages/admin-settings/SearchSettings.tsx` | surface readiness/ratio/per-type/capability | Modify |
|
||||
|
||||
---
|
||||
|
||||
## Tasks
|
||||
|
||||
### Task 1: Shared eligibility predicate, model+eligibility coverage counts, supporting index
|
||||
|
||||
**Files:** Create `internal/embeddingvectors/eligibility.go`, `migrations/sql/<ts>_media_item_embeddings_model_index.sql`; Modify `internal/recommendations/repo.go:150-160`, `internal/catalog/search_indexer.go:757-782` (+ callers `:133,:203,:310`, `search_service.go:139`); Test `internal/embeddingvectors/eligibility_test.go`, `internal/catalog/search_indexer_test.go`.
|
||||
|
||||
**Interfaces — Produces:**
|
||||
- `embeddingvectors.ItemEligibilityWhereClause(alias string) string`
|
||||
- `countCatalogSearchVectorDocuments(ctx, q coverageQuerier, itemTypes []string, model string) (int, error)` (adds `model`; applies `ELIG`)
|
||||
- `catalogSemanticCoverageByType(ctx, q coverageQuerier, itemTypes []string, model string) ([]catalogTypeCoverage, error)` → `{Type string; Eligible, Vectorized int}`
|
||||
|
||||
- [ ] Migration: `make migrate-create NAME=media_item_embeddings_model_index`; body uses `-- +goose NO TRANSACTION` then a self-healing guard for a leftover invalid index from a crashed build (follow the `20260618164519_add_episodes_season_id_index.sql` precedent: a `DO $$ ... NOT i.indisvalid ... DROP INDEX $$` block), then `CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_media_item_embeddings_model ON public.media_item_embeddings (model);` (down: `DROP INDEX CONCURRENTLY IF EXISTS`). The runner honors `NO TRANSACTION` (three shipping precedents). Run `make migrate-up`; verify with `make migrate-status`.
|
||||
- [ ] Move the eligibility predicate to `embeddingvectors`; refactor `recommendationItemEligibilityWhereClause` to delegate (assert output string unchanged).
|
||||
- [ ] Failing tests: (a) `model` filter excludes rows whose `e.model` differs; (b) an **unmatched, non-book** item inflates neither numerator nor denominator (the C1 regression guard); (c) denominator counts eligible items with no embedding row; (d) ratio ∈ [0,1] when a stale embedding exists on a now-unmatched item.
|
||||
- [ ] Implement both queries with `ELIG`; merge per type in Go; update **all four** callers (pass active model where available, else `""`).
|
||||
- [ ] `GOWORK=off go test ./internal/embeddingvectors ./internal/catalog -run 'Eligib|Coverage' -count=1` → PASS. Commit.
|
||||
|
||||
### Task 2: Active embedding model from the lock
|
||||
|
||||
**Files:** Modify recommendations `Engine` (`engine.go`), `internal/catalog/search_provider.go`; Test `internal/recommendations/*_test.go`.
|
||||
|
||||
**Produces:** `recommendations: func (e *Engine) ActiveEmbeddingModel(ctx) (string, error)` (→ `GetEmbeddingLock`; `lock.Model` or `""`); `catalog: type CatalogSemanticModelProvider interface { ActiveEmbeddingModel(ctx context.Context) (string, error) }`.
|
||||
|
||||
- [ ] Test: lock present → its `Model`; no lock → `("", nil)`.
|
||||
- [ ] Implement; confirm `recEngine` satisfies both `CatalogSearchQueryVectorizer` (the real interface name, `search_provider.go:77`) and `CatalogSemanticModelProvider`.
|
||||
- [ ] Run recommendations tests → PASS. Commit.
|
||||
|
||||
### Task 3: Coverage tracker — querier seam, snapshot, hysteresis, single-flight refresher
|
||||
|
||||
**Files:** Create `internal/catalog/semantic_coverage.go`, `semantic_coverage_test.go`; Modify `internal/catalog/search_provider.go` (consts + gate interface).
|
||||
|
||||
**Produces:**
|
||||
```go
|
||||
const (
|
||||
semanticCoverageEnableRatio = 0.90
|
||||
semanticCoverageDisableRatio = 0.80
|
||||
semanticCoverageRefreshInterval = 2 * time.Minute
|
||||
)
|
||||
|
||||
// coverageQuerier is the seam so Refresh is unit-testable without a real *pgxpool.Pool.
|
||||
type coverageQuerier interface {
|
||||
QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
|
||||
Query(ctx context.Context, sql string, args ...any) (pgx.Rows, error)
|
||||
}
|
||||
|
||||
type catalogTypeCoverage struct { Type string; Eligible, Vectorized int; Ratio float64; Ready bool }
|
||||
type semanticCoverageSnapshot struct { PerType map[string]catalogTypeCoverage; Overall float64; Model string; UpdatedAt time.Time }
|
||||
|
||||
type SemanticCoverageGate interface { CoverageReady(itemTypes []string) (ready bool, reason string) }
|
||||
|
||||
type semanticCoverageTracker struct { /* q coverageQuerier; indexTypes []string; models CatalogSemanticModelProvider; mu sync.Mutex; snap atomic.Pointer[semanticCoverageSnapshot]; clock func() time.Time */ }
|
||||
func (t *semanticCoverageTracker) Refresh(ctx context.Context) error
|
||||
func (t *semanticCoverageTracker) CoverageReady(itemTypes []string) (bool, string) // nil-snapshot ⇒ (false, "coverage not yet computed")
|
||||
func (t *semanticCoverageTracker) Snapshot() *semanticCoverageSnapshot
|
||||
func (t *semanticCoverageTracker) Run(ctx context.Context) // ticker; returns on ctx.Done()
|
||||
```
|
||||
|
||||
- [ ] Test (no panic / fail-safe): `CoverageReady` on a freshly-constructed tracker (no `Refresh` yet) returns `(false, "coverage not yet computed")` and touches the querier **zero** times (querier stub fails the test if called).
|
||||
- [ ] Test (hysteresis): 0.85→not ready; →0.92 ready; →0.85 **stays** ready; →0.79 not ready.
|
||||
- [ ] Test (scope): `CoverageReady(["movie","series"])` false when `series` < threshold though `movie` ready; reason names `series`. Empty scope requires all snapshot types ready.
|
||||
- [ ] Test (model collapse): snapshot computed for model A; active model becomes B → next `Refresh` publishes all-not-ready before recomputing.
|
||||
- [ ] Test (`-race`): concurrent `Refresh` + `CoverageReady` loops; assert monotonic `UpdatedAt`, no race.
|
||||
- [ ] Test (error retention): a `Refresh` whose query errors retains the prior snapshot.
|
||||
- [ ] Implement single-flight mutex, fresh-map immutability, nil-safe reads, model-collapse, retain-last-good, ticker `Run`.
|
||||
- [ ] `GOWORK=off go test ./internal/catalog -run Coverage -race -count=1` → PASS. Commit.
|
||||
|
||||
### Task 4: Wire gate + model provider; comma-ok degradation; start refresher in NewRouter
|
||||
|
||||
**Files:** Modify `internal/catalog/search_meilisearch_provider.go:27-66,329-334` (`MeilisearchProviderConfig` opens at `:27`), `internal/catalog/search_service.go:31-74`, `internal/api/router.go:437-552`.
|
||||
|
||||
**Consumes:** `SemanticCoverageGate`, `CatalogSemanticModelProvider`. **Produces:** `MeilisearchProviderConfig.Coverage SemanticCoverageGate`; `func (s *CatalogSearchService) StartCoverageRefresh(ctx context.Context)`.
|
||||
|
||||
- [ ] `shouldUseSemanticSearch` → `SemanticEnabled && wordCount>=3 && (p.config.Coverage == nil || ready)`; not-ready yields fallback reason `"semantic_not_ready: " + reason` (a **diagnostic, not an error**). Keep the `SkipTotal`-builds-hybrid test green.
|
||||
- [ ] In `NewCatalogSearchServiceFromSettings`: only when `SemanticEnabled`, derive the model provider via **comma-ok** `mp, ok := queryVectorizer.(CatalogSemanticModelProvider)`. If `!ok` or `queryVectorizer == nil` (semantic enabled but recommendations disabled — a real config), build the tracker with a model provider that yields `""` ⇒ every type not-ready (reason `"embedding model provider unavailable"`); **never** assert without comma-ok (nil-interface assertion panics). Store the tracker on the service; pass it as `Coverage`.
|
||||
- [ ] `StartCoverageRefresh` runs `tracker.Run(ctx)` once; no-op when there is no tracker (postgres/semantic-off). Call it from `NewRouter` after the service is built: `if catalogSearchService != nil { catalogSearchService.StartCoverageRefresh(deps.AppContext) }`. (The service is local to `NewRouter` and not returned; `deps.AppContext` is the process-lifetime ctx — Baseline #8/#9.)
|
||||
- [ ] Test: stub gate not-ready ⇒ long query stays keyword-only; ready gate + vectorizer ⇒ hybrid emitted; nil vectorizer + semantic-enabled ⇒ not-ready, no panic.
|
||||
- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit.
|
||||
|
||||
### Task 5: Diagnostics plumbing (provider → resolver → handler), correctly scoped
|
||||
|
||||
**Files:** Modify `internal/catalog/search_provider.go:64-71`, `internal/catalog/search_meilisearch_provider.go:189-237`, `internal/catalog/catalog_resolver.go:24-30,269-297`, `internal/api/handlers/catalog.go:55-61,91-134`; Test `search_provider_test.go`, `catalog_test.go`.
|
||||
|
||||
**Produces:** `CatalogSearchResult{+Mode string; +SemanticUsed bool}`; `CatalogResult{+Provider,Mode string; +SemanticUsed bool; +FallbackReason string}`; handler `type searchDiagnostics struct { Provider, Mode string; SemanticUsed bool; FallbackReason string \`json:"fallback_reason,omitempty"\` }`; `catalogResponse{+SearchDiagnostics *searchDiagnostics \`json:"search_diagnostics,omitempty"\`}`.
|
||||
|
||||
- [ ] Provider: `Mode="hybrid", SemanticUsed=true` only when the hybrid request is issued **and survives**; on the downgrade (`:221-228`) set `Mode="keyword", SemanticUsed=false, FallbackReason=semanticFallback`.
|
||||
- [ ] Resolver: copy the four fields in `resolveDirectSearchSource` (direct path only).
|
||||
- [ ] Handler: populate `search_diagnostics` only on the non-grouped direct path; **omit** for `group=work` and for non-relevance-sort `q=` requests.
|
||||
- [ ] Tests: (a) hybrid error ⇒ `semantic_used:false, mode:"keyword", fallback_reason` set (post-downgrade truth); (b) existing `total/total_exact/has_more/items/snapshot` byte-stable; (c) field **absent** for browse, for `group=work`, and for a `q=`+title-sort request.
|
||||
- [ ] `GOWORK=off go test ./internal/catalog ./internal/api/handlers -count=1` → PASS. Commit.
|
||||
|
||||
### Task 6: Capability validation + rate-limited probe; admin status from snapshot
|
||||
|
||||
**Files:** Modify `internal/catalog/search_meilisearch_client.go:167+`, `internal/catalog/search_meilisearch_provider.go`, `internal/catalog/search_service.go:90-145`, `internal/catalog/search_provider.go` (status structs).
|
||||
|
||||
**Produces:**
|
||||
```go
|
||||
func (c *meilisearchClient) GetSettings(ctx, uid string) (meilisearchIndexSettings, error) // GET /indexes/{uid}/settings
|
||||
type meilisearchIndexSettings struct { Embedders map[string]meilisearchEmbedderSettings `json:"embedders"` }
|
||||
type meilisearchEmbedderSettings struct { Source string `json:"source"`; Dimensions int `json:"dimensions"` }
|
||||
type CatalogSearchSemanticStatus struct {
|
||||
Ready bool `json:"ready"`; DisabledReason string `json:"disabled_reason,omitempty"`
|
||||
CoverageRatio float64 `json:"vector_coverage_ratio"`; CoverageUpdatedAt *time.Time `json:"coverage_updated_at,omitempty"`
|
||||
PerType []CatalogSearchTypeCoverage `json:"per_type,omitempty"`; Capability CatalogSearchSemanticCapability `json:"capability"`
|
||||
}
|
||||
type CatalogSearchTypeCoverage struct { Type string `json:"type"`; Eligible int `json:"eligible"`; Vectorized int `json:"vectorized"`; CoverageRatio float64 `json:"vector_coverage_ratio"`; Ready bool `json:"ready"` }
|
||||
type CatalogSearchSemanticCapability struct { OK bool `json:"ok"`; Reason string `json:"reason,omitempty"`; Embedder string `json:"embedder,omitempty"`; Dimensions int `json:"dimensions,omitempty"` }
|
||||
const semanticCapabilityProbeTTL = 5 * time.Minute
|
||||
```
|
||||
|
||||
- [ ] Add `Semantic CatalogSearchSemanticStatus` to `CatalogSearchRuntimeStatus`. **Keep** `document_count`/`vector_document_count` unchanged (additive only). The `Semantic` block (ratio, per-type, ready) is read from `tracker.Snapshot()` — **no fresh query** in `Status()`; if no tracker, `Ready=false, DisabledReason="semantic disabled"`.
|
||||
- [ ] Capability: embedder present, `source=="userProvided"`, `dimensions==embeddingvectors.CanonicalDimensions`; each failure → distinct `Reason`. Probe (unit vector, `limit:1`, `Embedder=config.Embedder`) only when `ActiveIndexUID != ""`, cached `semanticCapabilityProbeTTL`.
|
||||
- [ ] Tests: missing embedder / wrong source / wrong dimensions / probe failure each distinct; provider `Healthy` and circuit untouched in all four.
|
||||
- [ ] `GOWORK=off go test ./internal/catalog -count=1` → PASS. Commit.
|
||||
|
||||
### Task 7: Backfill cheap/expensive split with fairness
|
||||
|
||||
**Files:** Modify `internal/recommendations/repo.go:472-621`, `internal/recommendations/similar.go:251-362`; Test `repo_test.go`, `similar_test.go`.
|
||||
|
||||
**Produces:** `buildMissingOrModelStaleEmbeddingIDsSQL() string`; `ListMissingOrModelStaleEmbeddingIDs(ctx, afterID, currentModel string, limit int) ([]string, error)`; `BuildEmbeddingTextForIDs(ctx, ids []string) ([]EmbeddingTextCandidate, error)`; `EmbedAll` two-pass loop with dual cursors and an unconditional `embeddingTextStaleQuotaPerRun`.
|
||||
|
||||
- [ ] Test (SQL shape): `buildMissingOrModelStaleEmbeddingIDsSQL()` contains neither `"item_people"` nor `"LATERAL"`.
|
||||
- [ ] Test (behavior, no real DB needed for the shape test; behavior tests use existing recommendations test seams): an item that is *only* canonical-text-stale (people changed, model current) is **not** returned by the cheap query; a missing/model-stale item **is**.
|
||||
- [ ] Test (parity): `BuildEmbeddingTextForIDs(ids)` reproduces the same `canonical_text` as the legacy full CTE for the same ids, regardless of `ids` order.
|
||||
- [ ] Test (no starvation): with a permanently non-empty cheap backlog, a text-stale item is still embedded within a bounded number of `EmbedAll` runs (quota is unconditional); both cursors stay monotonic.
|
||||
- [ ] Implement; cheap `afterID` advances from the cheap query's last id.
|
||||
- [ ] `GOWORK=off go test ./internal/recommendations -count=1` → PASS. Commit.
|
||||
|
||||
### Task 8: Admin UI surfacing
|
||||
|
||||
**Files:** Modify `web/src/hooks/queries/admin/settings.ts:21-54`, `web/src/pages/admin-settings/SearchSettings.tsx:202-261`.
|
||||
|
||||
- [ ] Extend the `CatalogSearchStatus` TS type with the `semantic` block (ready, disabled_reason, vector_coverage_ratio, coverage_updated_at, per_type, capability).
|
||||
- [ ] Render a "Semantic readiness" row (ready/disabled + reason), overall coverage %, a per-type coverage table, a capability badge, and a "coverage updated" timestamp. Keep "Vectorized Documents" (`vector_document_count`).
|
||||
- [ ] `cd web && pnpm run lint && pnpm run format:check` → PASS. Commit.
|
||||
|
||||
---
|
||||
|
||||
## Test Plan
|
||||
|
||||
**Go unit (`internal/embeddingvectors`, `internal/catalog`):**
|
||||
- Eligibility predicate single-sourced; unmatched non-book item excluded from numerator **and** denominator (C1 guard); ratio ∈ [0,1].
|
||||
- Hybrid omitted when gate not ready; emitted when ready + capable + long query + vectorizer present.
|
||||
- **Model-stale safety:** only-old-model embeddings ⇒ coverage ~0 ⇒ keyword-only; model-change collapse publishes not-ready within one interval.
|
||||
- Hysteresis latch; mixed-scope AND; empty-scope = all-types; nil-snapshot returns not-ready with **no panic** and **no DB call**; `-race` refresh/read; error retention.
|
||||
- Capability: missing embedder / wrong source / wrong dimensions / probe failure each distinct; keyword stays up.
|
||||
|
||||
**Go unit (`internal/recommendations`):**
|
||||
- Cheap SQL excludes `item_people`/`LATERAL`; people-only-stale row excluded from cheap pass; per-ID text build matches legacy output order-independently; no text-stale starvation; dual cursors monotonic.
|
||||
|
||||
**Go unit (`internal/api/handlers`):**
|
||||
- `search_diagnostics` present on direct path; **absent** for browse, `group=work`, and non-relevance-sort `q=`; `semantic_used` reflects post-downgrade reality; existing fields byte-stable.
|
||||
|
||||
**Admin status:** `semantic.ready`, `disabled_reason`, `vector_coverage_ratio`, `per_type`, `capability`, `coverage_updated_at` sourced from the snapshot (no fresh query).
|
||||
|
||||
**Focused command:**
|
||||
```bash
|
||||
GOWORK=off go test ./internal/embeddingvectors ./internal/catalog ./internal/api/handlers ./internal/recommendations -race -count=1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Descoped — not in this pass
|
||||
|
||||
**Routing legacy `/search?q=` through the provider pipeline.** Sunset **Wed, 01 Jul 2026** (`internal/api/handlers/legacy_read_routes.go:10`, `legacyReadSunset`) — days away; hardening a path scheduled for deletion is negative ROI. The typed legacy path already routes through the resolver; only the bare-`q` fallback hits `itemRepo.Search`, which the sunset removes. Re-routing also risks `browseResponse`↔`catalogResponse` shape drift. Reopen as a one-task follow-up **only if the sunset slips**, with a byte-for-byte `browseResponse` assertion.
|
||||
|
||||
**After-sync coverage refresh trigger.** The indexer (`main.go:1730`) and the search service (`router.go:457`) share no handle, so an after-drain trigger would need new cross-component wiring and would race the interval refresh. The 2-minute interval plus model-change collapse is sufficient; the trigger is intentionally omitted.
|
||||
|
||||
---
|
||||
|
||||
## Assumptions
|
||||
|
||||
- **One additive index migration** (`media_item_embeddings(model)`, `CONCURRENTLY`). No schema/column changes. *(Revised from the original "no migration": the model-filtered numerator is only cheap with this index, and performance is priority #1.)*
|
||||
- All runtime coverage state is **in-memory** (atomic per-type snapshot), rebuilt on boot and refreshed every 2 minutes; not persisted.
|
||||
- Coverage is computed **entirely from Postgres** over the **embed-eligible** population (`status='matched' OR audiobook/ebook`), model-filtered, manga excluded, restricted to configured index types; it **never** uses the Meili `document_count`.
|
||||
- Active embedding model comes from `recommendations.EmbeddingLock.Model`; **no lock / no model provider ⇒ semantic not ready.**
|
||||
- A model change is reflected within ≤ one refresh interval and **fails safe** within that window via the model-collapse rule; the residual ≤2-minute mismatch window is acceptable for a rare, admin-gated reconfiguration.
|
||||
- Thresholds (enable 0.90 / disable 0.80) and the 2-minute interval are named consts and starting heuristics; the diagnostics shipped here are the instrument for retuning them. Promoting them to settings is a possible follow-up, not this pass.
|
||||
- `semantic_ratio` stays `0.30` (`DefaultMeilisearchSemanticRatio`).
|
||||
- Scope is hybrid-search hardening only — not conversational search.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review (writing-plans checklist)
|
||||
|
||||
- **Findings coverage:** C1 eligibility denominator (D1, Task 1) · C2 empty-scope semantics (D3, Task 3) · missing `model` index (D1 migration, Task 1) · 4-caller signature change (Baseline #4, Task 1) · refresher wiring in `NewRouter`/`AppContext` (Baseline #9, Task 4) · concurrent-refresh single-flight + immutability + `-race` (D2, Task 3) · nil-snapshot/error fail-safe (D2, Task 3) · model-change collapse (D2, Task 3) · comma-ok nil vectorizer (Task 4) · querier seam + SQL-string helper for testability (Task 3, Task 7) · grouped/non-relevance diagnostics scoping (D4, Task 5) · unit probe vector + admin-only/5m (D5, Task 6) · `Status()` reads snapshot (Task 6) · backfill cursors + fairness (D6, Task 7) · legacy & after-sync descoped (Descoped) — all mapped.
|
||||
- **Type consistency:** `SemanticCoverageGate`/`CatalogSemanticModelProvider`/`coverageQuerier`/`semanticCoverageTracker`/`catalogTypeCoverage`/`CatalogSearchSemanticStatus` used identically across tasks; `Eligible` (not `Indexable`) is the denominator field name throughout.
|
||||
- **No placeholders:** every new symbol has a signature; SQL is concrete; the migration body is specified.
|
||||
@@ -0,0 +1,248 @@
|
||||
# Unified Literary Works Design
|
||||
|
||||
## Goal
|
||||
|
||||
Silo should expose a server-owned literary work identity that links multiple format-specific catalog items for the same book, especially ebooks and audiobooks, without collapsing those items into one `media_items` row. Clients can then render one detail page with format tabs/actions while existing item detail, playback, reading, download, and catalog APIs keep working.
|
||||
|
||||
The implementation should target the ebook stack head, currently `work/ebook-reader-ruler-profiles`, which includes the ebook foundation, #98 local-cover/grouping work, and #99 reader profile/ruler work. That stack already has ebook scanning, ebook series metadata, ebook reader progress, and ebook detail extensions. The work layer should sit above those existing format-specific systems.
|
||||
|
||||
## Existing Context
|
||||
|
||||
`media_items.content_id` is the stable identity for each catalog item. Audiobooks are `media_items.type = 'audiobook'`; ebooks are introduced as `media_items.type = 'ebook'` in the ebook stack. These identities must remain independent because each format has different files, progress, playback/reader behavior, external compatibility, and library membership.
|
||||
|
||||
Audiobooks already store author and narrator credits in `item_people`, series data in `audiobook_series`, file data in `media_files`, and listening progress in existing playback/watch progress tables. The ebook stack adds ebook author credits, `ebook_series`, `ebook_reader_progress`, reader state/config, and ebook file serving. #98 also groups sibling ebook file formats by book identity during scan; that is an intra-ebook item grouping behavior, not the cross-format work identity.
|
||||
|
||||
The current catalog detail endpoint is `GET /api/v1/catalog/items/{id}` through `CatalogResourceHandler.HandleGetItemDetail`, backed by `catalog.DetailService.GetItemDetail`. Catalog browse/search remains item-oriented by default through catalog query/browse code.
|
||||
|
||||
## Data Model
|
||||
|
||||
Add a neutral work layer:
|
||||
|
||||
```sql
|
||||
CREATE TABLE literary_works (
|
||||
work_id TEXT PRIMARY KEY,
|
||||
canonical_title TEXT NOT NULL,
|
||||
sort_title TEXT,
|
||||
normalized_title TEXT NOT NULL,
|
||||
primary_author_key TEXT NOT NULL DEFAULT '',
|
||||
primary_cover_content_id TEXT REFERENCES media_items(content_id) ON DELETE SET NULL,
|
||||
description TEXT,
|
||||
published_date DATE,
|
||||
publisher TEXT,
|
||||
genres TEXT[] NOT NULL DEFAULT '{}',
|
||||
metadata JSONB NOT NULL DEFAULT '{}'::jsonb,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE literary_work_items (
|
||||
work_id TEXT NOT NULL REFERENCES literary_works(work_id) ON DELETE CASCADE,
|
||||
content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
|
||||
format_type TEXT NOT NULL,
|
||||
link_source TEXT NOT NULL,
|
||||
confidence DOUBLE PRECISION NOT NULL DEFAULT 1,
|
||||
confirmed_at TIMESTAMPTZ,
|
||||
ignored_at TIMESTAMPTZ,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (work_id, content_id),
|
||||
UNIQUE (content_id),
|
||||
CHECK (format_type IN ('ebook', 'audiobook', 'comic', 'manga')),
|
||||
CHECK (link_source IN ('manual', 'external_id', 'metadata_match', 'series_match', 'scan_seed'))
|
||||
);
|
||||
|
||||
CREATE TABLE literary_work_match_decisions (
|
||||
source_content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
|
||||
target_content_id TEXT NOT NULL REFERENCES media_items(content_id) ON DELETE CASCADE,
|
||||
decision TEXT NOT NULL,
|
||||
created_by INTEGER REFERENCES users(id) ON DELETE SET NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (source_content_id, target_content_id),
|
||||
CHECK (source_content_id <> target_content_id),
|
||||
CHECK (decision IN ('confirmed', 'ignored'))
|
||||
);
|
||||
```
|
||||
|
||||
Every ebook or audiobook can belong to at most one work through `UNIQUE (content_id)`. A work can contain many format-specific items. Manual/admin actions update `literary_work_items` and `literary_work_match_decisions`; automatic matching must not override an ignored pair or move a manually linked item without an explicit admin request.
|
||||
|
||||
Work IDs should be generated by the existing server ID generator, not derived directly from title text. Matching keys can change when metadata improves, but the `work_id` remains stable.
|
||||
|
||||
## Matching Rules
|
||||
|
||||
Matching runs in a `literaryworks` service after scan/enrichment or as an admin-triggered repair job. It considers literary item types only: `ebook`, `audiobook`, and later `comic`/`manga`.
|
||||
|
||||
Authority order:
|
||||
|
||||
1. Manual confirmed links and manual unlinks/ignored pairs.
|
||||
2. Shared external IDs from `media_item_provider_ids`, excluding format-specific aliases that are not true work identity.
|
||||
3. Normalized title plus normalized author.
|
||||
4. Normalized series name plus series index plus author.
|
||||
5. Publication year/date and publisher as supporting evidence only.
|
||||
|
||||
Automatic matching should be conservative. Same title alone is not enough. Same title plus different primary author should not link. Same title plus same author can link unless there is strong conflicting series/index evidence. Same series/index plus same author can link even if titles differ slightly due subtitle/edition formatting. Narrator is never required for ebook matching and must not prevent an ebook/audiobook work link; narrator remains audiobook-format metadata.
|
||||
|
||||
The matcher should produce scored candidates. Scores at or above the automatic threshold can create links when neither item has conflicting manual state. Lower scores are exposed as possible matches for admin review.
|
||||
|
||||
## Service Boundaries
|
||||
|
||||
Create an `internal/literaryworks` package with:
|
||||
|
||||
- `Repository`: reads/writes `literary_works`, memberships, decisions, and candidate rows.
|
||||
- `Matcher`: normalizes item metadata, scores candidates, and applies conservative automatic links.
|
||||
- `Service`: public orchestration used by handlers and background jobs.
|
||||
|
||||
The package depends on catalog repositories and the database pool, but catalog detail should depend on a small interface so item detail can include work data without creating a large import cycle.
|
||||
|
||||
The matcher should reuse existing item metadata:
|
||||
|
||||
- Titles from `media_items`.
|
||||
- Authors from `item_people.kind = author`.
|
||||
- Audiobook narrators from `item_people.kind = narrator` only for display inside audiobook format details.
|
||||
- Series from `audiobook_series` and `ebook_series`.
|
||||
- External IDs from `media_item_provider_ids`.
|
||||
- File metadata from `media_files`.
|
||||
|
||||
## API Contract
|
||||
|
||||
Add a work endpoint:
|
||||
|
||||
`GET /api/v1/works/{work_id}`
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"work_id": "work_123",
|
||||
"work_title": "Project Hail Mary",
|
||||
"authors": [{ "name": "Andy Weir", "person_id": "42" }],
|
||||
"formats": [
|
||||
{
|
||||
"type": "ebook",
|
||||
"content_id": "ebook_1",
|
||||
"library_id": 1,
|
||||
"available_files": [
|
||||
{
|
||||
"file_id": 10,
|
||||
"original_filename": "Project Hail Mary.epub",
|
||||
"format": "epub",
|
||||
"mime_type": "application/epub+zip",
|
||||
"size": 12345
|
||||
}
|
||||
],
|
||||
"progress": {
|
||||
"kind": "reading",
|
||||
"progress": 0.42,
|
||||
"updated_at": "2026-06-09T12:00:00Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "audiobook",
|
||||
"content_id": "audio_1",
|
||||
"library_id": 2,
|
||||
"available_files": [
|
||||
{
|
||||
"file_id": 20,
|
||||
"original_filename": "Project Hail Mary.m4b",
|
||||
"format": "m4b",
|
||||
"mime_type": "audio/mp4",
|
||||
"size": 987654321,
|
||||
"duration_seconds": 58200
|
||||
}
|
||||
],
|
||||
"progress": {
|
||||
"kind": "listening",
|
||||
"position_seconds": 1200,
|
||||
"duration_seconds": 58200,
|
||||
"updated_at": "2026-06-09T12:00:00Z"
|
||||
}
|
||||
}
|
||||
],
|
||||
"primary_cover_url": "/api/v1/images/...",
|
||||
"metadata": {
|
||||
"description": "...",
|
||||
"series": { "name": "The Series", "index": 1 },
|
||||
"genres": ["Science Fiction"],
|
||||
"published_date": "2021-05-04",
|
||||
"publisher": "Ballantine Books"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Extend `GET /api/v1/catalog/items/{content_id}` additively:
|
||||
|
||||
```json
|
||||
{
|
||||
"content_id": "ebook_1",
|
||||
"type": "ebook",
|
||||
"work_id": "work_123",
|
||||
"work_title": "Project Hail Mary",
|
||||
"work_formats": [
|
||||
{ "type": "ebook", "content_id": "ebook_1", "library_id": 1 },
|
||||
{ "type": "audiobook", "content_id": "audio_1", "library_id": 2 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Existing fields and behavior remain unchanged. Clients that do not know about works can ignore the new fields.
|
||||
|
||||
## Catalog Grouping
|
||||
|
||||
Add optional work grouping for literary media:
|
||||
|
||||
`GET /api/v1/catalog?type=reading&group=work`
|
||||
|
||||
The grouped response should return one card per work when a linked work exists, with format badges such as `ebook` and `audiobook`. Unlinked literary items still appear as single-item cards with an implicit one-item work projection. Non-literary media remain item-based.
|
||||
|
||||
Do not change default catalog/search behavior. Work grouping must be opt-in so current clients and Jellyfin/ABS compatibility routes are unaffected.
|
||||
|
||||
## Progress and Files
|
||||
|
||||
Progress remains per format and per existing content identity:
|
||||
|
||||
- Ebook progress reads from `ebook_reader_progress`.
|
||||
- Audiobook progress reads from existing playback/watch progress storage.
|
||||
|
||||
The work response aggregates those format-specific progress rows without merging them.
|
||||
|
||||
Available files come from `media_files`, grouped by format content ID. File metadata should include `file_id`, original basename, extension/container, MIME type when known, byte size, and duration for audio. Android can continue using existing file/download routes by `file_id`; the work endpoint only tells it what is available.
|
||||
|
||||
## Admin Primitives
|
||||
|
||||
Add admin endpoints:
|
||||
|
||||
- `GET /api/v1/admin/literary-works/items/{content_id}/matches`: list possible work/item matches with score and evidence.
|
||||
- `POST /api/v1/admin/literary-works/link`: link two or more content IDs into one work.
|
||||
- `POST /api/v1/admin/literary-works/{work_id}/unlink`: unlink one content ID from a work.
|
||||
- `POST /api/v1/admin/literary-works/matches/ignore`: mark a candidate pair ignored.
|
||||
- `POST /api/v1/admin/literary-works/matches/confirm`: confirm a candidate pair and link them.
|
||||
|
||||
Admin linking must validate that every content ID is a supported literary type and accessible to the admin. Unlinking the last item can delete the empty work.
|
||||
|
||||
## Backward Compatibility
|
||||
|
||||
All existing item detail, catalog, reader, playback, download, Jellyfin compatibility, and ABS compatibility routes remain valid. The new tables cascade from `media_items`, so library deletion and orphan cleanup continue to remove format items; work cleanup should delete empty works after membership deletion.
|
||||
|
||||
No caller is required to supply `work_id` for existing format actions. Work APIs are additive and read-oriented except for admin management endpoints.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Repository and matcher tests should cover:
|
||||
|
||||
- Ebook and audiobook with same normalized title and author link to one work.
|
||||
- Same title with different authors does not auto-link.
|
||||
- Same author and series/index links despite title subtitle differences.
|
||||
- Shared external ID links above fuzzy metadata.
|
||||
- Manual ignored pair blocks automatic linking.
|
||||
- Manual confirmed link wins over automatic scoring.
|
||||
- Unlink removes one item without deleting the format media item.
|
||||
- Existing item detail JSON still includes normal fields when no work exists.
|
||||
- Existing catalog browse/search remains item-based unless `group=work`.
|
||||
- Work endpoint returns separate ebook and audiobook progress.
|
||||
- Work endpoint returns file metadata for EPUB/PDF/CBZ and M4B/MP3 without changing download behavior.
|
||||
|
||||
Handler tests should assert route status codes and response shape. SQL tests should include cascade/empty-work cleanup behavior.
|
||||
|
||||
## Rollout Plan
|
||||
|
||||
Implement the schema and service behind additive APIs first. Then wire automatic matching after scan/enrichment. Finally add grouped catalog support. This keeps default behavior stable while giving clients and admin tooling a safe way to adopt works.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Server-side Kindle→EPUB conversion (MOBI / AZW / AZW3) — design
|
||||
|
||||
**Goal:** Let the silo-server render Kindle-family ebooks (mobi, azw, azw3) in the Android in-app reader by converting them to EPUB **server-side, in-process, on demand**, served transparently through the existing read endpoint. No external converter binary, no cgo.
|
||||
|
||||
**Status:** Approach approved by the user: convert via **libmobi**, run **built-in** to the server as **WASM executed by `wazero`** (a pure-Go runtime). On-demand + cached, admin-gated, DRM-free only. This is the MOBI half of the "all 9 server formats in-app" goal; the CBR (client-side) half is a separate Android spec.
|
||||
|
||||
> **Linchpin PROVEN (build spike, 2026-06-17, linux/amd64).** libmobi + `mobitool` cross-compile to `wasm32-wasi` (wasi-sdk 25, libmobi `9062742`, zlib 1.3.1 built to wasm); `-e` (create EPUB) is present; the module converts real MOBI6 / KF8-hybrid / HUFF-CDIC / unicode samples to **well-formed EPUB** (`mimetype`=`application/epub+zip` first, `META-INF/container.xml`, `OEBPS/*`). Verified end-to-end under **wazero** (the prod runtime) with WASI preopen + argv + `_start` + exit handling. Reproducible build stage committed at `tools/mobitool-wasm/Dockerfile` (with a smoke-conversion gate). Build-config gotcha resolved: build libmobi against a **wasm-compiled zlib** — `--with-zlib=no` makes libmobi vendor miniz that collides with mobitool's own miniz at `wasm-ld`. **DRM finding:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0** and writes a (garbage) EPUB — so detect failure by capturing stdout + validating output, never by exit code alone.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- Pure-Go MOBI/KF8 *parsers* don't exist at usable quality (the Go MOBI packages only *write*). Calibre is the gold standard but ~300MB+ of Python/Qt — rejected as too heavy. **libmobi** (mature C library behind the `mobitool` CLI) handles MOBI6 + KF8/AZW3 + AZW, decompression, and EPUB generation.
|
||||
- Rather than shell out to an external `mobitool` binary (a deploy moving part) or bind via cgo (forces `CGO_ENABLED=1`, a C toolchain, painful cross-compile, and an LGPLv3 static-link obligation), we compile **`mobitool` to a WASI `.wasm`** once and **run it in-process with `wazero`**.
|
||||
- This keeps the main Go build **`CGO_ENABLED=0`** and pure; one **architecture-independent** `.wasm` runs on amd64/arm64 servers alike; the `.wasm` is `go:embed`-ed into the binary (self-contained); libmobi's LGPL is cleanly isolated as a separate embedded artifact; and — a real bonus — **untrusted ebook bytes are parsed inside the WASM sandbox**, so a memory-safety bug in libmobi on a malicious file cannot reach the host (only the mapped temp dir).
|
||||
|
||||
## Components
|
||||
|
||||
### 1. The WASM module (build-time artifact)
|
||||
- Compile libmobi + its `mobitool` tool to `wasm32-wasi` using the **wasi-sdk** (clang). Output: `mobitool.wasm`.
|
||||
- Build libmobi **without libxml2** (`--with-libxml2=no` / `--disable-xmlwriter` as available) so the module has no external native deps; mobitool's EPUB writer uses a bundled miniz, which is portable C and compiles to WASI cleanly. (Feasibility of the no-libxml2 build for the `-e` path is the #1 thing to verify in the build spike — see Risks.)
|
||||
- Build it in a **dedicated Docker build stage** (or a committed `Makefile`/script under `tools/mobitool-wasm/`). **Commit the resulting `mobitool.wasm`** into the repo so the normal Go build does not need the wasi-sdk — it only `go:embed`s the prebuilt artifact. Rebuild the `.wasm` only when bumping libmobi; record the libmobi version + build flags next to it.
|
||||
|
||||
### 2. `internal/ebookconvert` (new package) — the in-process converter
|
||||
```go
|
||||
// Converter converts a Kindle-family ebook file to EPUB bytes, in-process via wazero.
|
||||
type Converter interface {
|
||||
// Convert reads srcPath (mobi/azw/azw3) and returns the path to a converted .epub.
|
||||
// Returns ErrDRMProtected / ErrUnsupported / ErrConversionFailed as appropriate.
|
||||
Convert(ctx context.Context, srcPath string) (epubPath string, err error)
|
||||
}
|
||||
```
|
||||
- `//go:embed mobitool.wasm` → compile the module **once** at startup with `wazero.NewRuntime` + `CompileModule` (the compiled module is cached and reused; only cheap instantiation happens per conversion).
|
||||
- **wazero command-module specifics** (from Codex review): `wasi_snapshot_preview1.MustInstantiate` the host module once; `Runtime.CompileModule` once at startup; per conversion instantiate with `ModuleConfig.WithArgs(...)` (argv ≈ `mobitool -e -o <outdir> <input>`), `WithFSConfig`, `WithStdout`+`WithStderr` (capture **both** — `mobitool.c` prints many failures with `printf` to stdout, not stderr), and the default `_start`. Use `WithName("")` (or a unique name) for repeated instantiation. Non-zero exits surface as `*sys.ExitError`. Honor ctx timeout via `RuntimeConfig.WithCloseOnContextDone(true)`.
|
||||
- **Filesystem sandboxing** (from Codex review): wazero's `WithDirMount`/`WithReadOnlyDirMount` still permit relative `../../` traversal *within host permissions*, so **never preopen the library directory or the cache root**. Per conversion: copy the source into a private scratch tree, mount input **read-only via an `fs.FS`** (`WithFSMount(os.DirFS(in), …)` — confined, no `..` escape) and a dedicated empty **writable** output dir (`WithDirMount`). **The WASM sandbox is a memory-safety boundary, not a hard filesystem jail** — a writable dir mount can still be traversed by a compromised guest within the server user's reach. Filesystem isolation therefore depends on a **deployment requirement: run silo-server as a constrained, non-root service user (the container already does), so the converter's blast radius is minimal.** This is documented here and in `internal/ebookconvert/converter.go`.
|
||||
- Enforce a wall-clock **timeout** (ctx), a **max source size**, and a **concurrency semaphore** (N simultaneous conversions). WASM memory is bounded by wazero config. **Resource note:** the defaults are `DefaultMaxMemoryPages` = 1 GiB linear memory per conversion × `DefaultConcurrency` = 2, i.e. up to ~2 GiB of wasm memory under saturation (plus the 256 MiB source / 512 MiB output caps). Generous for a small server; tune `ebookconvert.Options` for tight deployments.
|
||||
- Map mobitool's exit code / captured stdout+stderr to typed errors. **Deterministic** verdicts (cacheable): DRM-protected input → `ErrDRMProtected`; corrupt / unconvertible layout / oversize-output → `ErrConversionFailed`. **Transient** verdicts (never negatively cached, retried next read): the per-call timeout firing → `ErrConversionTimedOut`; source over the size cap → `ErrSourceTooLarge`; caller cancellation/deadline → the context error propagated verbatim.
|
||||
|
||||
### 3. Conversion cache (on-demand, keyed)
|
||||
- Cache converted EPUBs on disk under a configured cache root, keyed by **strong source identity + converter identity** — `file_id + size + mtime_ns + content hash` (use the scanner checksum if one exists) **plus** the converter **module fingerprint** (`moduleVersion`: a sha256 of the embedded `mobitool.wasm`, which already pins the libmobi commit, build flags, and wasi-sdk used to produce it). (Plain `path|size|mtime` can false-hit when a file is replaced with preserved mtime/size — Codex.) A stale key misses and reconverts.
|
||||
- **Singleflight** per key so concurrent reads of the same just-opened book convert once. The shared conversion runs on a **detached context** (decoupled from any one caller), so a single caller cancelling its request never aborts the work for the others.
|
||||
- **Negative caching**: remember only **deterministic-bad** sources — DRM (`ErrDRMProtected`) and conversion failures (`ErrConversionFailed`) — so every reader open does not reconvert a known-bad file. Transient outcomes (timeout, cancellation, oversize source) are **not** cached so they retry.
|
||||
- A cache **hit refreshes the entry's mtime**, so the mtime-ordered budget eviction acts as a real LRU; eviction never deletes the entry it is handing back or another conversion's in-flight temp.
|
||||
- Bounded by an LRU / total-size budget with opportunistic eviction (mirrors the existing reader-file/image cache conventions in the codebase).
|
||||
|
||||
### 4. Wiring into the read path
|
||||
- `serveEbookInline` (`internal/api/handlers/ebook_reader.go:602`) currently `os.Open`s `file.FilePath` and `http.ServeContent`s it with `ebookMimeType(...)`.
|
||||
- Change: when the source format is **kindle-family AND conversion is enabled**, resolve the converted EPUB via the cache (convert-on-miss), then serve **that** file with `Content-Type: application/epub+zip`. All other formats are served exactly as today (raw, byte-range, unchanged).
|
||||
- Byte-range/`http.ServeContent` semantics are preserved — we serve the cached EPUB `*os.File` the same way, so resume/range still works.
|
||||
- **Response headers** (from Codex review): `Content-Type: application/epub+zip`, `Content-Disposition` with an `.epub` filename, a strong `ETag` derived from the conversion cache key, and — because the *same URL* can return raw or converted bytes depending on the setting/failure — `Cache-Control: private, max-age=0, must-revalidate` (or `no-store`) to prevent stale client/proxy caching of the wrong representation.
|
||||
- **HEAD is cache-only.** A `HEAD /read` must never trigger a (possibly minute-long, ~1 GiB) conversion. It answers from the cache: a hit serves the real converted headers + `Content-Length`; a negatively-cached source serves the `failed` contract; a miss advertises the converted headers (`Content-Type`, the cache-key `ETag`, `X-Silo-Ebook-Conversion: converted`) **optimistically with no body and no conversion** — the subsequent `GET` produces the body and the authoritative DRM/failure verdict, which then populates the cache that later HEADs read. *(Client note: a HEAD miss intentionally omits `Content-Length`; coordinate with the Android/Apple clients if any relies on it.)*
|
||||
|
||||
### 5. Admin gate + client capability
|
||||
- An **admin-toggleable setting** (`ebook.kindle_conversion_enabled`, default off) gates the whole feature, via the existing server settings mechanism (`internal/api/handlers/settings.go` / `admin.go` `HandleUpdateSetting`). The flag is read with a short TTL cache so the hot read path (and the capability endpoint clients poll) does not hit the DB per request, while toggles still apply within seconds without a restart. When off: kindle-family files serve raw as today and the capability is not advertised.
|
||||
- **Two gates to advertise the capability** (from Codex review): advertise conversion availability to clients **only when** (a) the admin flag is on **and** (b) the embedded WASM module compiled + instantiated successfully at startup. The Android client flips mobi/azw/azw3 from `ExternalOnly`→`InApp` **only when the flag is advertised**; otherwise it keeps today's external-open fallback. (Android-side change is small and tracked separately; this spec is server-side.)
|
||||
- **Capability endpoint (implemented):** `GET /api/v1/ebooks/capability` → `{enabled, source_formats:["mobi","azw","azw3"], served_format:"epub", header:"X-Silo-Ebook-Conversion", header_failed_value:"failed"}`. `enabled` is true only when both gates hold (`EbookConversion.active`). The read response carries `X-Silo-Ebook-Conversion: converted|failed` so the client knows whether the body is EPUB or the raw original (→ external-open). ETag on the converted response is derived from the **exact conversion cache key** (`SourceKey.CacheKey()`) so a validator can never disagree with the cache; the failed/raw fallback is `Cache-Control: no-store` since the same URL flips representations with the flag.
|
||||
|
||||
## Data flow (read of a `.azw3`, conversion enabled)
|
||||
1. `GET /api/v1/ebooks/{cid}/files/{fid}/read` → authorize → `file` is `azw3`, conversion enabled.
|
||||
2. Cache lookup by source identity+version. Hit → serve cached `.epub`. Miss → singleflight: `Converter.Convert` (wazero runs `mobitool.wasm` on a temp copy) → store under key.
|
||||
3. `http.ServeContent` the `.epub` with `application/epub+zip`, byte-range supported.
|
||||
4. Client sees EPUB → existing reflow `EpubReader` renders it. Reader progress stays keyed to the **source** `fileId` (the EPUB is a derived view of the same file), so progress/annotations are unaffected.
|
||||
|
||||
## Error handling — explicit client failure contract
|
||||
Transparent raw fallback is **not** safe on its own: once the server advertises conversion and the Android in-app EPUB path opens `/read`, returning raw MOBI/AZW bytes hands the EPUB reader a non-EPUB body (Codex). So failures need a contract the client can detect:
|
||||
- On conversion failure (DRM, corrupt, timeout, oversize) the read endpoint returns a **structured signal** the Android client maps to its external-open fallback — preferred: a distinct status (e.g. `422 Unprocessable` / `415`) with an error code, **or** serve raw bytes **plus** an explicit `X-Silo-Ebook-Conversion: failed` (with the original MIME) header that the client checks. Pick one and make both sides honor it. Never silently hand raw bytes to the EPUB path.
|
||||
- `ErrDRMProtected` vs `ErrConversionFailed` are distinguished in logs + (optionally) the error code, so DRM books get the "open externally" message and transient failures can be retried.
|
||||
- Module compile/instantiate failure at startup (corrupt embed) → capability **not advertised** (gate b), kindle files serve raw as today, logged loudly.
|
||||
|
||||
## Testing
|
||||
- `internal/ebookconvert` unit tests with **small DRM-free MOBI6, AZW, and KF8/AZW3 fixtures**: `Convert` produces a valid EPUB (zip with `mimetype` first + `META-INF/container.xml` + OPF + at least one content doc), deterministic across runs; corrupt input → `ErrConversionFailed`; oversize → rejected before invocation; timeout honored (ctx cancel).
|
||||
- Cache tests: miss→convert→hit; key changes on source mtime/size and on module-version bump; singleflight collapses concurrent conversions.
|
||||
- Handler tests: enabled+kindle → `application/epub+zip` body is a valid EPUB; disabled → raw bytes + original MIME; non-kindle → unchanged; DRM/failed → graceful fallback path.
|
||||
- A build-stage check (CI) that the committed `mobitool.wasm` matches a rebuild from the pinned libmobi (or at least that it instantiates + converts a fixture), so the artifact can't silently rot.
|
||||
|
||||
## Sequencing — spike first (Codex-recommended)
|
||||
**The first PR is only the build spike + a tiny Go harness that converts fixtures** — no handler/cache/wiring until the linchpin is proven. Codex confirmed libmobi documents both deps as optional (`--with-zlib=no` → bundled `miniz.c`; `--with-libxml2=no` → internal xmlwriter) and that `mobitool.c` gates `-e` behind `USE_XMLWRITER`, not `HAVE_LIBXML2` — so the no-libxml2 wasm build is *plausible*, but the spike must **assert** `mobitool -h` exposes `-e` and a fixture actually converts. Only then design the handler/cache/wiring PR.
|
||||
|
||||
## Risks / spike-first items
|
||||
1. **libmobi+mobitool compiles to `wasm32-wasi` with the `-e` (EPUB) path working without libxml2.** The linchpin. If libxml2 turns out mandatory for `-e`, fall back to compiling libxml2 to wasm too (heavier) or assembling the EPUB in Go from libmobi's `MOBIRawml` parts.
|
||||
2. **wazero WASI preopen file IO** for the file sizes involved (tens of MB) — confirm mapping + memory/time limits.
|
||||
3. mobitool CLI surface as a wasm **command module** (argv + preopens) — confirm wazero runs it via `_start` and we capture exit status + stdout + stderr.
|
||||
|
||||
## Committed `mobitool.wasm` — guardrails (Codex)
|
||||
Commit the artifact (so normal Go builds need no wasi-sdk) **with**: the pinned libmobi source commit, the Dockerfile/build script that produced it, its SHA256, the build flags, and the LGPL license notice. Add a **CI smoke test** that instantiates the embedded module and converts a fixture. Do not gate CI on byte-for-byte rebuild reproducibility — `mobitool -v` embeds `__DATE__`/`__TIME__`, so a rebuild won't match byte-for-byte unless normalized; rely on the fixture-conversion smoke + metadata hash instead.
|
||||
|
||||
## Process gate (Codex)
|
||||
`docs/architecture/v1-scope.md` indicates v1 scope is not locked and new user-facing capabilities may need a proposal / project-tracking entry **before** a feature PR. Confirm this process step with the maintainer before opening the conversion PR. *(Decision deferred to the user.)*
|
||||
|
||||
## Out of scope
|
||||
- CBR (separate Android client-side spec).
|
||||
- DRM removal (never).
|
||||
- Converting at scan/ingest time (on-demand only).
|
||||
- Surfacing a separate "EPUB version" in the catalog (we serve transparently through `/read`).
|
||||
- Reusing the WASM converter for other formats (later, if useful).
|
||||
|
||||
## Commit / deploy rules (project)
|
||||
- silo-server changes go on a **feature branch, committed locally, PR-ready — NOT pushed** (per standing rule; server changes land via PR).
|
||||
@@ -0,0 +1,711 @@
|
||||
# Offline sync for mobile (downloads v2) — design
|
||||
|
||||
**Goal:** Let mobile clients (Android, Apple) download media for **fully offline** playback and keep
|
||||
watch state consistent across reconnects. Three pillars: (1) a **device-scoped download registry** —
|
||||
the server tracks what each device has downloaded; (2) an **offline playback manifest** — one bundle
|
||||
giving the client everything it needs to play with no network (metadata, subtitle files, artwork,
|
||||
chapter + intro/credits markers); (3) **offline progress reconciliation** — the client queues watch
|
||||
progress while offline and the server merges it on reconnect (last-write-wins). Downloads are requested
|
||||
through an admin-gated **quality** ladder (`original`, then bitrate presets), with server-side
|
||||
**transcode-to-a-single-file** for bitrate requests and compatibility fallback for devices that cannot
|
||||
play the source directly.
|
||||
|
||||
**Status:** Approved — implementing. This capability cleared the Silo v1 scope gate on
|
||||
2026-06-18 (via the **v1 capability proposal** triage; template
|
||||
`.github/ISSUE_TEMPLATE/v1-capability-proposal.yml`), and the offline-sync/downloads-v2 work is now
|
||||
being implemented phase by phase against this design. Per the decision below, the
|
||||
`/api/v1/downloads` contract is **reshaped, not just extended** — acceptable because the change lands
|
||||
**before** scope lock (`docs/architecture/v1-scope.md`) and **before** any mobile client ships against
|
||||
it. This document began as the design input to that proposal and now serves as the implementation
|
||||
reference; it is kept in sync as the implementation lands.
|
||||
|
||||
Decisions locked with the requester:
|
||||
- **Replace** the existing `internal/download` package with a new unified package (`internal/downloads`)
|
||||
that absorbs today's download behavior and adds the three pillars.
|
||||
- **The existing `/api/v1/downloads` contract may be reshaped, not just extended.** The only current
|
||||
consumer is the first-party web app, which is updated in lockstep. So downloads v2 is a single
|
||||
coherent surface: one device-aware `downloads` table and one `/downloads/*` endpoint family — *not* a
|
||||
frozen legacy table plus a parallel registry. (This intentionally diverges from the v1 additive-only
|
||||
API rule, which is acceptable here because the change lands **before** scope lock and **before** any
|
||||
mobile client ships against it — exactly the right time to reshape. The proposal must call this out so
|
||||
reviewers/clients know the download contract changed pre-lock.)
|
||||
- **Quality is user-facing; delivery format is internal.** Clients request `quality` from the ordered
|
||||
ladder `original > 20mbps > 10mbps > 5mbps > 2mbps > 1mbps`. `remux` is not a preset; it is a
|
||||
server-selected `delivery_format` when `original` needs container/audio compatibility. Bitrate
|
||||
presets are gated by a server setting **and** the per-user `download_transcode_allowed` flag.
|
||||
- **Transcode-to-single-file is in v1.** (Today's transcode pipeline only emits ephemeral HLS segments.)
|
||||
- **No expiry, no lease, no DRM.** Downloaded files persist on-device until the user deletes them; the
|
||||
server can only revoke *future* downloads via `users.download_allowed`.
|
||||
- **No cross-device visibility** in v1 (a device sees its own entries, not other devices').
|
||||
|
||||
**Implementation update (2026-06-20):** the current client-facing contract is documented in
|
||||
`docs/downloads-api.md`. The public request field is `quality`, capabilities expose
|
||||
`quality_presets`, rows expose `quality`, `effective_quality`, `delivery_format`,
|
||||
`target_bitrate_kbps`, and `revision`, and batch manifests are implemented at
|
||||
`GET /api/v1/downloads/batches/{batch_id}/manifests`.
|
||||
|
||||
> Commands and paths in this document are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- **One device-aware table, not two.** The original plan kept today's `downloads` table
|
||||
(`migrations/sql/042_downloads.sql`, keyed on `user_id`) frozen and added a separate device registry,
|
||||
purely to avoid breaking the existing contract. With that constraint lifted, the cleaner model is a
|
||||
single `downloads` table reshaped to carry `device_id` (and `profile_id`). `device_id` is **nullable**:
|
||||
- `device_id IS NULL` → an **ephemeral / account-level** download (today's web-app browser download:
|
||||
pick a version, pull the file; the row is a convenience record, not a managed library entry).
|
||||
- `device_id` set → a **managed device-library entry** (a phone "has" this file; durable; unique per
|
||||
`(user, profile, device, content, episode)`; the target of manifest + progress reconciliation).
|
||||
One table, one endpoint family, and the format/transcode machinery is shared by web and mobile alike.
|
||||
- **Device identity already exists in Postgres.** `migrations/sql/180_user_devices.sql` defines
|
||||
`public.user_devices` keyed `(user_id, profile_id, device_id)` with FKs to `users` and
|
||||
`user_profiles`. Device id arrives in the `X-Silo-Device-Id` header (see
|
||||
`internal/api/handlers/settings.go:37`). Managed entries reference it; ephemeral rows leave it NULL.
|
||||
- **The last-write-wins primitive already exists.** `userstore.UserStore.SetProgressIfNewer(... updatedAt) (bool, error)`
|
||||
(`internal/userstore/store.go:26`, implemented in `internal/userdb/progress.go:131`) is exactly the
|
||||
merge semantics offline reconciliation needs. Progress reconciliation is therefore **additive changes
|
||||
to the existing progress endpoints**, not new machinery, and it deliberately stays in the watch-state
|
||||
domain rather than being folded into the downloads package.
|
||||
- **Manifests must never carry presigned URLs.** `catalog.ItemDetail` (`internal/catalog/detail.go`)
|
||||
exposes `PosterURL`/`BackdropURL` as short-lived presigned S3 URLs — useless for an offline bundle
|
||||
that may be opened days later. The manifest carries **stable** references (thumbhashes inline +
|
||||
authenticated proxy endpoints the client fetches once at download time), so a stored manifest never
|
||||
expires.
|
||||
- **Remux and transcode both need a produced file, not a live pipe.** A clean, seekable, resumable
|
||||
download requires a finalized file (`-movflags +faststart` relocates the moov atom in a finalization
|
||||
pass — impossible over a pure pipe). `internal/playback`'s current `ServeRemux`/`StartTranscode`
|
||||
stream ephemeral output. So remux and transcode share one **prepare-to-file** job pipeline; only
|
||||
`original` is served straight from the source on disk.
|
||||
|
||||
## The replacement: `internal/downloads` (new package)
|
||||
|
||||
`internal/download` (singular) is replaced by `internal/downloads` (plural). The new package owns the
|
||||
whole downloads domain; everything below lives in it unless noted.
|
||||
|
||||
```
|
||||
internal/downloads/
|
||||
model.go — Download (web + device, one type), Artifact, OfflineManifest,
|
||||
quality/delivery constants, status constants, sentinel errors
|
||||
repo.go — Postgres CRUD: downloads (reshaped), download_artifacts
|
||||
service.go — orchestration: permission + quota checks, quality policy, create/list/serve
|
||||
policy.go — DownloadQualityResolver (quality ladder + delivery decision + server/user gates)
|
||||
bandwidth.go — BandwidthManager (ported from internal/download/bandwidth.go)
|
||||
limiter.go — QuantityLimiter (ported from internal/download/limiter.go)
|
||||
manifest.go — ManifestBuilder (assembles OfflineManifest; strips all presigned URLs)
|
||||
artifacts.go — ArtifactManager (async prepare-to-file jobs: remux + transcode, dedup, cleanup)
|
||||
serve.go — file serving for original source and prepared artifacts
|
||||
```
|
||||
|
||||
**What it absorbs / reshapes** from `internal/download`: the `downloads` table and its endpoints, now
|
||||
reshaped to be device-aware and quality/delivery-aware; `BandwidthManager`; `QuantityLimiter`; the `Service`
|
||||
permission/quota/file-serving logic; and the `config.DownloadConfig` integration.
|
||||
|
||||
**What stays out of the package (by design):**
|
||||
- **Progress reconciliation** is watch-state, not downloads. It is additive changes to
|
||||
`internal/api/handlers/progress.go` + `internal/userstore`/`internal/userdb`, reusing
|
||||
`SetProgressIfNewer`. Folding it into `internal/downloads` would couple two domains and duplicate the
|
||||
LWW write.
|
||||
- **The encode/remux primitive** stays in `internal/playback` (it owns ffmpeg). `internal/downloads`
|
||||
calls a new `playback.PrepareFile(...)`; it does not shell out to ffmpeg itself.
|
||||
|
||||
**Cutover strategy:** because the web app is the only consumer and is updated in lockstep, Phase 0 is a
|
||||
*reshape*, not a verbatim port: move the logic into `internal/downloads`, reshape the table + endpoints
|
||||
(add device-awareness plus quality/delivery fields), update the web app's download hooks/components, delete
|
||||
`internal/download`. The handler `internal/api/handlers/downloads.go` already depends only on a narrow
|
||||
`DownloadService` interface, so the swap is mechanical; its request/response DTOs gain the new fields.
|
||||
|
||||
## Data model (Goose migrations)
|
||||
|
||||
Migrations are timestamped, created with `make migrate-create NAME=...` (never hand-numbered; never
|
||||
`goose fix`).
|
||||
|
||||
### 1. Reshape `downloads` (device-aware, format-aware)
|
||||
|
||||
A single migration alters the existing table. Downloads ship default-off with little/no production data,
|
||||
so widening it in place is safe.
|
||||
|
||||
```sql
|
||||
-- +goose Up
|
||||
-- +goose StatementBegin
|
||||
ALTER TABLE public.downloads
|
||||
ADD COLUMN profile_id text, -- NULL for ephemeral/web rows
|
||||
ADD COLUMN device_id text, -- NULL = ephemeral; set = managed device entry
|
||||
ADD COLUMN format text NOT NULL DEFAULT 'original',
|
||||
ADD COLUMN artifact_id text; -- set for remux/transcode once prepared
|
||||
|
||||
ALTER TABLE public.downloads
|
||||
ADD CONSTRAINT downloads_format_check CHECK (format IN ('original','remux','transcode'));
|
||||
|
||||
-- Widen the status enum to cover the managed-entry lifecycle alongside the existing serve states.
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT downloads_status_check;
|
||||
ALTER TABLE public.downloads ADD CONSTRAINT downloads_status_check
|
||||
CHECK (status IN ('queued','downloading','completed','failed','cancelled', -- existing
|
||||
'registered','preparing','ready','revoked')); -- managed entries
|
||||
|
||||
-- One managed entry per (user, profile, device, content, episode). Ephemeral rows (NULL device_id)
|
||||
-- are exempt via the partial index.
|
||||
CREATE UNIQUE INDEX downloads_device_entry_uidx
|
||||
ON public.downloads (user_id, profile_id, device_id, content_id, COALESCE(episode_id, ''))
|
||||
WHERE device_id IS NOT NULL;
|
||||
|
||||
CREATE INDEX downloads_device_idx ON public.downloads (user_id, profile_id, device_id) WHERE device_id IS NOT NULL;
|
||||
CREATE INDEX downloads_artifact_idx ON public.downloads (artifact_id) WHERE artifact_id IS NOT NULL;
|
||||
|
||||
-- Composite FK to user_devices. MATCH SIMPLE (the default) skips the check whenever any column is NULL,
|
||||
-- so ephemeral rows (NULL device_id) are not constrained while managed entries are.
|
||||
ALTER TABLE public.downloads
|
||||
ADD CONSTRAINT downloads_device_fkey
|
||||
FOREIGN KEY (user_id, profile_id, device_id)
|
||||
REFERENCES public.user_devices(user_id, profile_id, device_id) ON DELETE CASCADE;
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- +goose StatementBegin
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_device_fkey;
|
||||
DROP INDEX IF EXISTS downloads_artifact_idx;
|
||||
DROP INDEX IF EXISTS downloads_device_idx;
|
||||
DROP INDEX IF EXISTS downloads_device_entry_uidx;
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_format_check;
|
||||
ALTER TABLE public.downloads DROP CONSTRAINT IF EXISTS downloads_status_check;
|
||||
ALTER TABLE public.downloads ADD CONSTRAINT downloads_status_check
|
||||
CHECK (status IN ('queued','downloading','completed','failed','cancelled'));
|
||||
ALTER TABLE public.downloads
|
||||
DROP COLUMN artifact_id, DROP COLUMN format, DROP COLUMN device_id, DROP COLUMN profile_id;
|
||||
-- +goose StatementEnd
|
||||
```
|
||||
|
||||
Status lifecycle:
|
||||
- **ephemeral / web** (`device_id IS NULL`): `queued → downloading → completed` / `failed` /
|
||||
`cancelled` — exactly today's behavior, now also able to request `remux`/`transcode`.
|
||||
- **managed entry** (`device_id` set):
|
||||
- `original`: `registered → ready` immediately (client pulls the source), then client reports
|
||||
`completed`.
|
||||
- `remux` / `transcode`: `registered → preparing` (artifact job runs) `→ ready → completed`; or
|
||||
`failed`.
|
||||
- `revoked`: admin/permission revocation of *future* access. Cannot delete the on-device file (no DRM
|
||||
by decision); it only stops the server serving new bytes.
|
||||
|
||||
The unique index uses `COALESCE(episode_id,'')` so movies (NULL `episode_id`) collapse to one managed
|
||||
entry per `(device, content)` without NULL-comparison surprises.
|
||||
|
||||
### 2. `download_artifacts` — prepared (remux/transcode) files, deduplicated
|
||||
|
||||
```sql
|
||||
-- +goose Up
|
||||
-- +goose StatementBegin
|
||||
CREATE TABLE public.download_artifacts (
|
||||
id text NOT NULL,
|
||||
media_file_id integer NOT NULL REFERENCES public.media_files(id) ON DELETE CASCADE,
|
||||
format text NOT NULL, -- 'remux' | 'transcode'
|
||||
params_hash text NOT NULL, -- sha256 of the encode parameters (see below)
|
||||
container text NOT NULL DEFAULT 'mp4',
|
||||
codec_video text NOT NULL DEFAULT '',
|
||||
codec_audio text NOT NULL DEFAULT '',
|
||||
resolution text NOT NULL DEFAULT '',
|
||||
audio_track_index integer NOT NULL DEFAULT -1,
|
||||
output_path text NOT NULL DEFAULT '', -- absolute path on the server's artifact volume
|
||||
file_size bigint NOT NULL DEFAULT 0,
|
||||
status text NOT NULL DEFAULT 'queued',
|
||||
error_message text NOT NULL DEFAULT '',
|
||||
-- Durable-queue / crash-recovery columns (see "Durable artifact queue" below).
|
||||
attempts integer NOT NULL DEFAULT 0,
|
||||
max_attempts integer NOT NULL DEFAULT 3,
|
||||
lease_owner text, -- worker/node id holding the running lease
|
||||
lease_expires_at timestamptz, -- NULL unless status='running'
|
||||
next_retry_at timestamptz, -- backoff gate for re-enqueue after a failure
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
completed_at timestamptz,
|
||||
last_used_at timestamptz NOT NULL DEFAULT now(),
|
||||
CONSTRAINT download_artifacts_pkey PRIMARY KEY (id),
|
||||
CONSTRAINT download_artifacts_unique UNIQUE (media_file_id, format, params_hash),
|
||||
CONSTRAINT download_artifacts_status_check CHECK (status IN ('queued','running','ready','failed'))
|
||||
);
|
||||
|
||||
CREATE INDEX download_artifacts_lru_idx ON public.download_artifacts (last_used_at) WHERE status = 'ready';
|
||||
-- Claimable work: queued rows whose backoff has elapsed, plus running rows whose lease has expired.
|
||||
CREATE INDEX download_artifacts_claimable_idx ON public.download_artifacts (status, next_retry_at);
|
||||
CREATE INDEX download_artifacts_lease_idx ON public.download_artifacts (lease_expires_at) WHERE status = 'running';
|
||||
-- +goose StatementEnd
|
||||
|
||||
-- +goose Down
|
||||
-- +goose StatementBegin
|
||||
DROP TABLE IF EXISTS public.download_artifacts;
|
||||
-- +goose StatementEnd
|
||||
```
|
||||
|
||||
`params_hash = sha256(format | container | codec_video | codec_audio | resolution | audio_track_index | subtitle_burn_in)`.
|
||||
Two devices (or a web user and a phone) requesting the same 1080p H.264/AAC of the same source share one
|
||||
artifact. `last_used_at` drives LRU cleanup; `output_path` is on a configured artifact volume. The
|
||||
`attempts`/`lease_*`/`next_retry_at` columns make the table a **durable, recoverable job queue** (see the
|
||||
"Durable artifact queue" subsection) so a crash mid-encode cannot strand a download in `preparing`.
|
||||
|
||||
### 3. Server settings (no DDL)
|
||||
|
||||
`server_settings` is key/value, so new keys need no migration — written when an admin saves them, parsed
|
||||
in `internal/config/db_loader.go`:
|
||||
- `download.transcode_enabled` (bool, default **false**) — server gate for transcode-to-file.
|
||||
- `download.artifact_dir` (string) — artifact output volume; defaults under the existing transcode/temp root.
|
||||
- `download.max_concurrent_prepares` (int, default 2) — encode/remux worker-pool size.
|
||||
- `download.artifact_max_bytes` (int64, default 0 = unlimited) — LRU eviction budget for artifacts.
|
||||
|
||||
`config.DownloadConfig` (`internal/config/config.go:299`) gains the matching fields; they hot-reload via
|
||||
the existing 30s-TTL cache (`internal/download/service.go:107` → ported), and the keys are registered as
|
||||
non-restart in `internal/config/restart_keys.go`.
|
||||
|
||||
## Quality policy
|
||||
|
||||
Resolved in `internal/downloads/policy.go` at create time, for both web and device flows:
|
||||
|
||||
```go
|
||||
const (
|
||||
QualityOriginal = "original"
|
||||
Quality20Mbps = "20mbps"
|
||||
Quality10Mbps = "10mbps"
|
||||
Quality5Mbps = "5mbps"
|
||||
Quality2Mbps = "2mbps"
|
||||
Quality1Mbps = "1mbps"
|
||||
)
|
||||
|
||||
type QualityDecision struct {
|
||||
RequestedQuality string
|
||||
EffectiveQuality string
|
||||
DeliveryFormat string // original | remux | transcode
|
||||
TargetBitrateKbps int
|
||||
RequiresArtifact bool
|
||||
}
|
||||
|
||||
func (DownloadQualityResolver) Resolve(requested string, user *models.User,
|
||||
cfg config.DownloadConfig, file *models.MediaFile, caps playback.ClientCapabilities,
|
||||
artifactsAvailable bool) (QualityDecision, error)
|
||||
```
|
||||
|
||||
Decision tree:
|
||||
1. Empty quality defaults to `original`.
|
||||
2. `original` direct-plays the source unless supplied device caps require a compatibility artifact.
|
||||
Container/audio-only compatibility records `delivery_format=remux`; video transcode fallback records
|
||||
`effective_quality=20mbps`, `delivery_format=transcode`.
|
||||
3. Bitrate presets require `cfg.TranscodeEnabled` (server), `user.DownloadTranscodeAllowed`, and an
|
||||
available artifact pipeline. Server gate off → `ErrTranscodeDisabled` (403). User flag off →
|
||||
`ErrDownloadNotAllowed` (403). Missing artifact pipeline → `ErrQualityUnavailable` (501).
|
||||
4. `remux` and `transcode` as request values are invalid public qualities.
|
||||
|
||||
Target codec/resolution for compatibility artifacts and bitrate transcodes is chosen by reusing `playback.Resolve` /
|
||||
`playback.SelectVersion` (`internal/playback/resolver.go:60`) against the device's declared capabilities
|
||||
and the admin transcode ceilings, so download encoding matches streaming decisions exactly (no
|
||||
duplicated codec logic).
|
||||
|
||||
## Transcode-to-file pipeline (v1)
|
||||
|
||||
`internal/downloads/artifacts.go` — `ArtifactManager`. Remux and transcode share this; `original` never
|
||||
enters it.
|
||||
|
||||
**On create of a remux/transcode download (web or device):**
|
||||
1. Resolve target params (codec/res/container/audio track) → compute `params_hash`.
|
||||
2. `INSERT ... ON CONFLICT (media_file_id, format, params_hash) DO NOTHING` into `download_artifacts`;
|
||||
read the row. If `status = ready`, link `downloads.artifact_id`, set `status = ready`,
|
||||
`file_size = artifact.file_size`, bump `last_used_at`. Done — no new encode.
|
||||
3. Otherwise the download row is `preparing`; if the artifact is freshly `queued`, enqueue an encode job.
|
||||
|
||||
**Encode worker** (bounded pool of `download.max_concurrent_prepares`, hosted on the existing
|
||||
`internal/taskmanager` worker machinery so jobs are tracked, observable in admin, and cancellable):
|
||||
1. **Claim a job transactionally** (see "Durable artifact queue" below) — never just "enqueue freshly
|
||||
queued rows," which loses work on restart.
|
||||
2. Call a new `playback.PrepareFile(ctx, opts)` that builds **single-file** ffmpeg args and writes to
|
||||
`output_path + ".part"`, then atomically renames to `output_path`:
|
||||
- **remux:** `-map ... -c copy -movflags +faststart -f mp4` (fast, near disk-throughput).
|
||||
- **transcode:** the existing encode args (reuse the `buildFFmpegArgs` logic / `TranscodeOpts` at
|
||||
`internal/playback/transcode.go:30`, incl. HW-accel, audio-track selection, optional subtitle
|
||||
burn-in) but targeting `-movflags +faststart -f mp4` to a file instead of `-f hls`.
|
||||
`TranscodeOpts` already carries every needed field (`InputPath`, `TargetCodecVideo/Audio`,
|
||||
`TargetResolution`, `HWAccel`, `AudioTrackIndex`, `SubtitleBurnIn`, `TotalDuration`).
|
||||
3. On success: artifact → `ready` (`file_size`/`completed_at`, clear lease); flip every linked download
|
||||
row → `ready`; publish a `user_state`/`downloads` event on the events hub (`internal/events`) so a
|
||||
waiting client pulls without polling.
|
||||
4. On failure: increment `attempts`, set `error_message`/`next_retry_at` (backoff). If
|
||||
`attempts >= max_attempts` → artifact `failed` and linked rows `failed`; otherwise return it to
|
||||
`queued` for retry. Either way, delete the stale `.part` file.
|
||||
|
||||
**Durable artifact queue (crash/restart recovery).** The artifact table is the queue of record, so a
|
||||
process exit mid-encode cannot strand a download in `preparing` forever:
|
||||
- **Transactional claim:** a worker claims a job with a single statement —
|
||||
`UPDATE download_artifacts SET status='running', lease_owner=$me, lease_expires_at=now()+$lease, attempts=attempts+1
|
||||
WHERE id = (SELECT id FROM download_artifacts
|
||||
WHERE (status='queued' AND (next_retry_at IS NULL OR next_retry_at <= now()))
|
||||
OR (status='running' AND lease_expires_at < now()) -- steal expired leases
|
||||
ORDER BY created_at LIMIT 1 FOR UPDATE SKIP LOCKED) RETURNING *`.
|
||||
`FOR UPDATE SKIP LOCKED` makes concurrent workers (and multiple nodes) safe without double-encoding.
|
||||
- **Lease heartbeat:** a running worker periodically extends `lease_expires_at`; if it dies, the lease
|
||||
expires and another worker (or the startup sweep) reclaims the job.
|
||||
- **Startup sweep:** on boot, reset `running` rows with an expired lease back to `queued` (bounded by
|
||||
`max_attempts`), and re-link any `preparing` download rows to their artifact's real state. A `ready`
|
||||
artifact whose `output_path` is missing on disk is reset to `queued` (the file was lost).
|
||||
- **Idempotency:** the encode writes to `output_path + ".part"` and atomically renames; a reclaimed job
|
||||
overwrites its own `.part`. `output_path` is deterministic from `(media_file_id, format, params_hash)`.
|
||||
- **No silent stranding:** a download stuck in `preparing` past a bound is surfaced (admin task + a
|
||||
`failed` transition with reason), never left invisibly hung.
|
||||
|
||||
**Serving** (`internal/downloads/serve.go`, `GET /downloads/{id}/file`):
|
||||
- `original`: open `media_files.file_path`, `http.ServeContent` (Range/resume) wrapped in the ported
|
||||
`BandwidthManager.ThrottledReader`. Same path as today's `serveFileDownload`.
|
||||
- `remux`/`transcode`: require `status = ready`; serve `artifact.output_path` the same way (Range/resume
|
||||
+ throttle); bump `last_used_at`.
|
||||
|
||||
**Cleanup:** an admin/cron job (mirroring `internal/playback/transcode_cleanup.go`) evicts `ready`
|
||||
artifacts whose `last_used_at` is older than a retention window or when the total exceeds
|
||||
`download.artifact_max_bytes` (LRU), but never one still linked by an active managed download row,
|
||||
including completed rows that represent a device's local library state.
|
||||
Whatever is dropped is logged (no silent truncation).
|
||||
|
||||
## Offline playback manifest
|
||||
|
||||
`internal/downloads/manifest.go` builds `OfflineManifest` from `catalog`'s existing detail path (reusing
|
||||
`catalog.ItemDetail` assembly — markers, chapters, subtitle list, versions) and then **strips every
|
||||
presigned URL**. The manifest is stable: a client can store it and re-read it offline indefinitely.
|
||||
Manifest + asset endpoints are only meaningful for managed device entries (`device_id` set).
|
||||
|
||||
```go
|
||||
type OfflineManifest struct {
|
||||
DownloadID string `json:"download_id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
Type string `json:"type"` // movie | episode
|
||||
Revision int `json:"revision"`
|
||||
Quality string `json:"quality"`
|
||||
EffectiveQuality string `json:"effective_quality"`
|
||||
DeliveryFormat string `json:"delivery_format"`
|
||||
TargetBitrateKbps int `json:"target_bitrate_kbps"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
Runtime int `json:"runtime,omitempty"`
|
||||
ContentRating string `json:"content_rating,omitempty"`
|
||||
Genres []string `json:"genres,omitempty"`
|
||||
SeriesID string `json:"series_id,omitempty"`
|
||||
SeriesTitle string `json:"series_title,omitempty"`
|
||||
SeasonNumber *int `json:"season_number,omitempty"`
|
||||
EpisodeNumber *int `json:"episode_number,omitempty"`
|
||||
|
||||
// Artwork: stable thumbhashes inline (immediate offline placeholder) + authenticated proxy URLs
|
||||
// the client downloads ONCE at download time. Never presigned S3 URLs.
|
||||
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
|
||||
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
|
||||
ArtworkURLs struct {
|
||||
Poster string `json:"poster,omitempty"`
|
||||
Backdrop string `json:"backdrop,omitempty"`
|
||||
Logo string `json:"logo,omitempty"`
|
||||
} `json:"artwork_urls"`
|
||||
|
||||
// Playback metadata
|
||||
Container string `json:"container"`
|
||||
CodecVideo string `json:"codec_video"`
|
||||
CodecAudio string `json:"codec_audio"`
|
||||
Resolution string `json:"resolution"`
|
||||
HDR bool `json:"hdr"`
|
||||
Duration int `json:"duration_seconds"`
|
||||
|
||||
Chapters []OfflineChapter `json:"chapters,omitempty"`
|
||||
Intro *Marker `json:"intro,omitempty"`
|
||||
Credits *Marker `json:"credits,omitempty"`
|
||||
Recap *Marker `json:"recap,omitempty"`
|
||||
Preview *Marker `json:"preview,omitempty"`
|
||||
|
||||
Subtitles []OfflineSubtitle `json:"subtitles"`
|
||||
|
||||
// Stable identity so the client can re-resolve content_id after a server-side rescan
|
||||
// (mirrors userstore.WatchIdentity at internal/userstore/types.go:130).
|
||||
StableIdentity OfflineIdentity `json:"stable_identity"`
|
||||
|
||||
ManifestVersion int `json:"manifest_version"` // bump on DTO shape change
|
||||
GeneratedAt string `json:"generated_at"` // RFC3339
|
||||
}
|
||||
|
||||
type OfflineSubtitle struct {
|
||||
Language string `json:"language"`
|
||||
Format string `json:"format"` // srt | ass | vtt
|
||||
Forced bool `json:"forced"`
|
||||
HearingImpaired bool `json:"hearing_impaired"`
|
||||
External bool `json:"external"`
|
||||
FetchURL string `json:"fetch_url"` // authenticated proxy; client downloads at download time
|
||||
FileSize int64 `json:"file_size,omitempty"`
|
||||
}
|
||||
```
|
||||
|
||||
**Artwork/subtitle proxies** (replace presigning):
|
||||
- `GET /downloads/{id}/artwork/{kind}` (`kind` ∈ `poster|backdrop|logo`) reads the raw stored path and
|
||||
**streams the bytes** through the existing image resolver instead of redirecting to a presigned URL.
|
||||
The client caches these once.
|
||||
- `GET /downloads/{id}/subtitles/{ref}` (`ref` encodes `external:{index}` or `downloaded:{id}`) streams
|
||||
the subtitle file (external sub from disk; downloaded sub from S3 via the existing subtitle manager).
|
||||
|
||||
Both are authorized under the **full managed-entry rule** (see "Security & reliability invariants"):
|
||||
the row must match `(user_id, profile_id, header device_id)`, and content/library access is **re-checked
|
||||
for the requesting profile** before any bytes are served (a download id alone never authorizes access).
|
||||
They are session-authenticated rather than time-limited tokens, so a manifest stored on-device stays valid.
|
||||
|
||||
## Progress reconciliation
|
||||
|
||||
Two **additive** changes. The naïve version (route the client's wall-clock straight into
|
||||
`SetProgressIfNewer` and drive `?since=` off the same column) is **unsafe**: a skewed or malicious device
|
||||
can submit a far-future timestamp, win LWW permanently over genuinely-newer server activity, and poison
|
||||
the cursor (a future-dated row makes a reader advance its cursor past real writes). So the design
|
||||
**separates client event time from server ordering**.
|
||||
|
||||
The watch-progress row gains two server-owned facets (LWW key vs. cursor ordering):
|
||||
- **`event_at`** — the client-observed wall-clock for the progress event, used as the **LWW comparison
|
||||
key**, but **bounded on ingest**: `event_at = min(client_updated_at, server_now + skew)` with a small
|
||||
skew window (e.g. 2m); a value past the window is clamped to `server_now` (and logged). For an online
|
||||
write `event_at = server_now`. Bounding means a client can at most claim "now" for its **own profile's**
|
||||
progress — which it is already entitled to do — and can never lock in a far-future value.
|
||||
- **`synced_seq`** — a **server-assigned monotonic** marker set on every write (a sequence, or
|
||||
server-ingest timestamp with a tie-breaker). It is **never** client-influenced and is the sole basis
|
||||
for the `?since=` cursor.
|
||||
|
||||
**Write — `POST /api/v1/sync/progress`** (`internal/api/handlers/progress.go:230`): add an optional
|
||||
`updated_at` (RFC3339) per item = the client event time. The handler clamps it (above) and merges via an
|
||||
extended `SetProgressIfNewer` whose comparison is on the **bounded `event_at`**, while the server stamps
|
||||
`synced_seq` itself. Absent `updated_at` → behavior is exactly as today (server `now()`), so existing
|
||||
callers (incl. watchsync) are unaffected. Completion is never inferred from the timestamp alone — the
|
||||
existing watched-threshold logic still gates `completed`.
|
||||
|
||||
```go
|
||||
type syncProgressItem struct {
|
||||
MediaItemID string `json:"media_item_id"`
|
||||
Position float64 `json:"position"`
|
||||
Duration float64 `json:"duration"`
|
||||
ForceOverwrite bool `json:"force_overwrite"`
|
||||
UpdatedAt *string `json:"updated_at,omitempty"` // NEW: RFC3339; client EVENT time (server clamps it)
|
||||
}
|
||||
```
|
||||
|
||||
**Read — `GET /api/v1/progress?since=<cursor>`** (`internal/api/handlers/progress.go:86`): the `since`
|
||||
cursor is an **opaque server token over `synced_seq`**, not a client timestamp. When present, only rows
|
||||
with `synced_seq > cursor` are returned (use a sequence, or `>=` + id de-dup if a timestamp marker is
|
||||
used, to avoid tie loss). The response carries the next cursor. This makes delta delivery immune to
|
||||
client clock skew. Requires `UserStore.ListProgressSince(ctx, profileID, cursor)` in both backends
|
||||
(`internal/userdb`, `internal/userstore/pgstore`) ordering/filtering on `synced_seq`. Absent `since` →
|
||||
identical to today.
|
||||
|
||||
Adding a `UserStore` method (and the `event_at`/`synced_seq` columns to the per-user `watch_progress`
|
||||
schema) touches every stub implementation; ship the implementations, the schema change, and the shared
|
||||
store test suite update (`internal/userstore/storetest`) in the same PR.
|
||||
|
||||
**Net safety:** honest clients still get correct "most-recent actual watch wins" LWW; a dishonest clock
|
||||
is bounded to claiming "now" for its own profile (no authority it didn't already have), and cursors —
|
||||
the cross-device delivery mechanism — never depend on a client clock at all.
|
||||
|
||||
## API surface (one `/downloads/*` family; existing endpoints reshaped)
|
||||
|
||||
The web app is updated in lockstep, so existing endpoints are reshaped rather than frozen.
|
||||
|
||||
**Reshaped (existing):**
|
||||
```
|
||||
POST /downloads body {content_id, episode_id?, file_id?, quality?, series?, caps?};
|
||||
X-Silo-Device-Id present → managed entry; absent → ephemeral/web row.
|
||||
`quality` is original by default; bitrate presets are single-item only.
|
||||
GET /downloads managed entries for the CALLING device (device from header); ephemeral rows otherwise
|
||||
DELETE /downloads/{id} remove a row owned by (user, profile, header device)
|
||||
GET /downloads/{id}/file serve original (source) or prepared artifact (Range/resume + throttle)
|
||||
GET|HEAD /direct-download one-shot browser download, original-only (browser-friendly, token-in-URL)
|
||||
```
|
||||
The response DTO gains `device_id`, `quality`, `effective_quality`, `delivery_format`,
|
||||
`target_bitrate_kbps`, `revision`, artifact-derived readiness, and the new statuses.
|
||||
**`device_id` is authoritative only from the `X-Silo-Device-Id` header — never from the body or query**
|
||||
(a body/query value could only ever be a display hint, and is not accepted as authority).
|
||||
|
||||
**Bitrate qualities are single-item only.** A series batch (`series:true`) and
|
||||
`/direct-download` are **original-quality only**: `/direct-download` streams
|
||||
synchronously and cannot wait on an async encode, and a series batch would otherwise
|
||||
fan out N encode jobs. A client that wants prepared transcodes for a series requests
|
||||
each episode individually. `DownloadQualityResolver` rejects bulk bitrate requests
|
||||
with `ErrBulkQualityUnavailable` / HTTP 501.
|
||||
|
||||
**New — capability / feature detection** (pattern from `internal/api/handlers/notifications.go:324`):
|
||||
```
|
||||
GET /downloads/capability
|
||||
→ { "enabled": bool, "download_allowed": bool,
|
||||
"quality_presets": ["original","20mbps","10mbps","5mbps","2mbps","1mbps"],
|
||||
"transcode_enabled": bool, "transcode_user_allowed": bool,
|
||||
"season_download": bool, "series_monitoring": bool }
|
||||
```
|
||||
|
||||
**New — managed-entry lifecycle + offline bundle:**
|
||||
```
|
||||
PATCH /downloads/{id} body {status:"downloading"|"completed"}
|
||||
GET /downloads/{id}/manifest → OfflineManifest
|
||||
GET /downloads/batches/{batch_id}/manifests → {manifests:[OfflineManifest]}
|
||||
GET /downloads/{id}/artwork/{kind} → image bytes (poster|backdrop|logo)
|
||||
GET /downloads/{id}/subtitles/{ref} → subtitle bytes
|
||||
```
|
||||
|
||||
**Progress reconciliation:** additive fields on existing `POST /sync/progress` and `GET /progress`
|
||||
(above). No new routes.
|
||||
|
||||
Routes mount in the existing authenticated group where the current download routes live
|
||||
(`internal/api/router.go:2121`). **Every managed-entry endpoint — create, list, `PATCH`, `DELETE`,
|
||||
`/file`, `/manifest`, `/artwork`, `/subtitles` — is profile-scoped (`RequireProfile`) and authorizes the
|
||||
row on `(user_id, profile_id, header device_id)`** (not `user_id` alone — household profiles share a
|
||||
`user_id`, so a user-only check would leak one profile's downloads to another). A managed call requires
|
||||
`X-Silo-Device-Id` (missing header on a managed create → `400 device_id_required`); ephemeral web rows
|
||||
omit it and stay account-scoped. Byte-serving and asset endpoints additionally **re-check per-profile
|
||||
content/library access** (the existing access filter / `EnsureAccessible`) before serving, so a stale or
|
||||
out-of-scope row — e.g. a child profile, or a profile that lost library access — cannot pull restricted
|
||||
media by download id even if the row exists.
|
||||
|
||||
## Data flows
|
||||
|
||||
**Ephemeral web download (reshaped existing flow, quality-aware)**
|
||||
```
|
||||
POST /downloads {content_id, quality:"original", caps:{...}} (no device id → ephemeral)
|
||||
→ Resolve quality/delivery · permission · quota
|
||||
→ compatibility artifact: ArtifactManager.Ensure → preparing|ready ; direct original: ready immediately
|
||||
client → GET /downloads/{id}/file → serveContent(source|artifact, Range, throttle)
|
||||
```
|
||||
|
||||
**Register a managed bitrate download (mobile)**
|
||||
```
|
||||
POST /downloads {content_id, quality:"5mbps", caps:{...}} (+X-Silo-Device-Id, +X-Profile-Id)
|
||||
→ Resolve: cfg.TranscodeEnabled && user.DownloadTranscodeAllowed → delivery_format=transcode
|
||||
→ upsert downloads (device entry, status=preparing)
|
||||
→ ArtifactManager.Ensure(media_file_id, params): ON CONFLICT dedup → enqueue encode if new
|
||||
encode worker → playback.PrepareFile(... -movflags +faststart) → rename → artifact=ready
|
||||
→ linked rows=ready → events hub notify
|
||||
client (on notify) → GET /downloads/{id}/file → serveContent(artifact, Range, throttle)
|
||||
→ GET /downloads/{id}/manifest → download ArtworkURLs.* and Subtitles[].FetchURL once
|
||||
→ PATCH /downloads/{id} {status:"completed"}
|
||||
```
|
||||
|
||||
**Offline progress, reconnect**
|
||||
```
|
||||
(offline) client queues {media_item_id, position, duration, updated_at} per stop
|
||||
(online) POST /sync/progress {items:[... updated_at ...]} → SetProgressIfNewer per item
|
||||
GET /progress?since=<cursor> → rows changed elsewhere since last sync
|
||||
```
|
||||
|
||||
## Phased build sequence (each phase independently shippable)
|
||||
|
||||
- **Phase 0 — Reshape + scaffolding (coordinated web update).** Create `internal/downloads`; reshape the
|
||||
`downloads` table (device plus quality/delivery columns, widened status) and the `/downloads` +
|
||||
`/direct-download` endpoints to be quality-aware; port `BandwidthManager`/`QuantityLimiter`/serving;
|
||||
extend `config.DownloadConfig` (+ new keys, default-off); add quality/delivery constants + sentinel
|
||||
errors; add `GET /downloads/capability`. Update the web app's download hooks/components
|
||||
(`web/src/hooks/queries/downloads.ts`, `web/src/components/DownloadVersionPicker.tsx`) to the reshaped
|
||||
DTOs. Delete `internal/download`.
|
||||
- **Phase 1 — Managed device entries.** Device-aware create/list/patch/delete; managed-entry serving for
|
||||
`original`; `X-Silo-Device-Id` handling + the device-entry unique constraint. **Invariant 2** lands
|
||||
here: profile+device authorization on every managed endpoint, header-only `device_id` authority.
|
||||
*Acceptance tests:* a second profile on the same `user_id` is denied a managed row's `/file` and
|
||||
`PATCH`/`DELETE`; a body/query `device_id` cannot override the header.
|
||||
- **Phase 2 — Offline manifest.** `ManifestBuilder`; manifest + artwork + subtitle proxy endpoints, each
|
||||
under the invariant-2 authorization + a per-profile content-access re-check before serving.
|
||||
*Acceptance test:* a child/library-restricted profile is denied artwork/subtitle/manifest for an
|
||||
out-of-scope download id.
|
||||
- **Phase 3 — Prepare-to-file (remux + transcode) [v1-critical].** `download_artifacts` migration (incl.
|
||||
lease/attempt columns); `ArtifactManager` as a **durable leased queue** (transactional claim, lease
|
||||
heartbeat, startup sweep, `.part` cleanup) per **invariant 3**; `playback.PrepareFile`; encode worker on
|
||||
`taskmanager`; ready-state serving; events notification; cleanup job. Turn `download.transcode_enabled`
|
||||
on. *Acceptance test:* kill the process mid-encode → on restart the artifact re-enqueues and the linked
|
||||
download reaches `ready` (no permanent `preparing`); concurrent workers never double-encode.
|
||||
- **Phase 4 — Progress reconciliation.** `event_at` (clamped) + server-assigned `synced_seq` columns on
|
||||
`watch_progress`; clamped `updated_at` on `POST /sync/progress`; server-token `?since=` on
|
||||
`GET /progress`; `ListProgressSince` in both stores + store test suite, per **invariant 1**.
|
||||
*Acceptance test:* a future-dated client `updated_at` is clamped (cannot win over a later real write)
|
||||
and never advances another device's cursor.
|
||||
|
||||
Phase 0 precedes all and is the coordinated server+web reshape. Phases 1/2/4 are independent; Phase 3
|
||||
depends on 1. Remux can ship in Phase 3 even if transcode stays admin-off.
|
||||
|
||||
## Client impact
|
||||
|
||||
- **Android (`silo-android`):** persist a stable `device_id` (UUID in app storage) and send
|
||||
`X-Silo-Device-Id`; call `/downloads/capability` on login to gate UI; register downloads, poll/listen
|
||||
for `ready`, pull the file, store the manifest + artwork + subtitle files locally; play the local file
|
||||
with ExoPlayer; queue offline progress with timestamps in a Room table and flush via
|
||||
`POST /sync/progress`, then pull deltas with `GET /progress?since=`.
|
||||
- **Apple (`silo-apple`):** same flow; `device_id` from `identifierForVendor` (or stored UUID on macOS);
|
||||
`URLSession` background download against `GET /downloads/{id}/file`; AVFoundation plays the local file;
|
||||
offline progress queue drained on reconnect. The manifest's markers/chapters reuse the types already
|
||||
used for online playback.
|
||||
- **Web (`web/`):** updated in lockstep in **Phase 0** — the download hooks/components move to the
|
||||
reshaped `/downloads` DTOs (quality/effective quality/delivery fields, new statuses). New admin additions: a
|
||||
`download.transcode_enabled` toggle in admin settings, plus the existing per-user
|
||||
`download_transcode_allowed` switch in `web/src/pages/AdminUserDetail.tsx`. No offline-library UI on web.
|
||||
|
||||
## Security & reliability invariants (must-hold)
|
||||
|
||||
These three are correctness requirements, not nice-to-haves; each ships with the acceptance test named in
|
||||
the phase plan. They were hardened in response to an adversarial design review.
|
||||
|
||||
1. **Server-owned sync ordering.** Cross-device delta delivery (`?since=`) is driven only by the
|
||||
server-assigned `synced_seq`; the client clock is bounded metadata (`event_at`, clamped to
|
||||
`now + skew`) used only for LWW conflict resolution on the caller's own profile. A client-supplied
|
||||
timestamp can never advance another reader's cursor or lock in a far-future "win."
|
||||
2. **Full profile + device authorization.** Every managed-entry endpoint authorizes on
|
||||
`(user_id, profile_id, header device_id)` and re-checks per-profile content/library access before
|
||||
serving bytes/assets. A download id is never sufficient; `device_id` authority is the header only.
|
||||
This holds the "no cross-device visibility" and "no cross-profile leakage" invariants given that
|
||||
household profiles share a `user_id`.
|
||||
3. **Durable artifact recovery.** Artifact preparation is a leased, attempt-counted, transactionally
|
||||
claimed queue with a startup sweep, so no crash/restart can strand a download in `preparing` or
|
||||
double-encode the same artifact.
|
||||
|
||||
## Risks & trade-offs
|
||||
|
||||
- **Coordinated server+web reshape (Phase 0).** Reshaping `/downloads` requires the web app to land in
|
||||
the same milestone. Mitigated because both are first-party and the reshape is mechanical (added fields,
|
||||
not removed behavior for the existing web path).
|
||||
- **One table, two lifecycles.** A nullable `device_id` plus a wide status enum carries both ephemeral
|
||||
and managed rows. Deliberate and managed (partial unique index + MATCH-SIMPLE FK keep the two modes'
|
||||
invariants clean); the alternative two-table split was only justified by the now-lifted compat
|
||||
constraint.
|
||||
- **Artifact disk + CPU cost.** Mitigated by `(media_file_id, format, params_hash)` dedup (shared across
|
||||
web/devices/users), a bounded encode pool, LRU/byte-budget cleanup, and transcode being **off by
|
||||
default** (admin opt-in).
|
||||
- **Remux is not free for a clean downloadable file.** `+faststart` needs a finalization pass, so even
|
||||
remux is a prepare-to-file job. It is fast (`-c copy`) but not instant; `preparing → ready` + push
|
||||
notification covers the UX.
|
||||
- **`content_id` can change on rescan.** The manifest carries `StableIdentity` (provider IDs +
|
||||
season/episode) mirroring `WatchIdentity` so the client re-resolves after a catalog rebind; the stale
|
||||
row doesn't break the on-device file.
|
||||
- **No expiry by decision.** A revoked `download_allowed` stops *future* serves (re-checked each serve,
|
||||
as today at `internal/download/service.go:345`) but cannot remove an already-downloaded file. Call this
|
||||
out in admin copy.
|
||||
- **LWW vs. client clocks (resolved, see invariant 1).** Conflict resolution uses a clamped `event_at`
|
||||
and delivery uses the server-owned `synced_seq`, so a skewed/malicious clock is bounded to "now" on its
|
||||
own profile and cannot poison cursors. Residual: an honest-but-wrong client clock within the skew
|
||||
window can still mis-order two of *its own* near-simultaneous events — acceptable.
|
||||
- **Authorization breadth (resolved, see invariant 2).** The per-endpoint profile+device authorization
|
||||
and pre-serve access re-check are load-bearing; the acceptance suite must include a cross-profile and a
|
||||
cross-device denial test, since a regression here is a content-access leak, not just a bug.
|
||||
- **`UserStore` gains a method + `watch_progress` gains columns.** `ListProgressSince` plus the
|
||||
`event_at`/`synced_seq` columns touch every store implementor/stub and the per-user schema — all in the
|
||||
Phase 4 PR.
|
||||
- **Quota semantics.** v1: all download rows (web + device) count toward the **same** per-user
|
||||
`QuantityLimiter`; revisit if too coarse.
|
||||
|
||||
## Out of scope (deferred)
|
||||
|
||||
Cross-device download visibility; download expiry/lease/DRM; cumulative per-user storage-byte quotas;
|
||||
push-to-client "your transcode is ready" beyond the existing events hub; server-initiated deletion of
|
||||
client-side files.
|
||||
|
||||
## Appendix — proposal-ready summary (maps to the v1 capability template)
|
||||
|
||||
- **Summary:** Mobile apps can download movies/episodes for fully offline playback — choosing original
|
||||
or an admin-gated bitrate quality — and watch progress stays consistent when the device reconnects.
|
||||
The web app's existing download feature is folded into the same reshaped contract. Remux remains an
|
||||
internal delivery format, not a client-facing quality preset.
|
||||
- **User value / why v1:** Offline playback is table-stakes for mobile media clients (commutes, flights,
|
||||
spotty networks) and the client teams need the download + sync contract locked to build against.
|
||||
Reshaping the existing download contract now — before lock and before any mobile client ships against
|
||||
it — is the right time; the only current consumer is the first-party web app, updated in lockstep.
|
||||
- **API surface:** the reshaped + new `/downloads/*` endpoints listed in "API surface" above (capability,
|
||||
device-aware create/list/patch, file serving, manifest/artwork/subtitle proxies) plus
|
||||
`updated_at`/`?since=` on the progress endpoints. **Note for reviewers:** this *changes* the existing
|
||||
`/downloads` contract (additive-only is intentionally waived pre-lock for this first-party endpoint).
|
||||
- **Client impact:** Android — full offline UI + sync; Apple — full offline UI + sync; Web — lockstep
|
||||
Phase-0 DTO update + admin toggles.
|
||||
- **Rough size (server-side):** **L** (new package replacing the old one, table reshape + artifacts
|
||||
table, async encode pipeline, manifest assembly, additive progress sync).
|
||||
+246
@@ -0,0 +1,246 @@
|
||||
# Personal Collection Catalog Filter Standardization - Design
|
||||
|
||||
- **Date:** 2026-06-23
|
||||
- **Status:** Draft (rev. 3 after adversarial review). Incorporates three findings: `match: any` merge safety, display-query fragment normalization, and ebook misclassification under unscoped watched filters.
|
||||
- **Scope:** `silo-server` backend and web collection UI. Client follow-ups in `silo-android` and `silo-apple` are optional unless they adopt the same collection editing surface.
|
||||
- **Note:** All paths are repository-relative. Commands assume the repository root is the cwd.
|
||||
|
||||
## Current Branch State
|
||||
|
||||
This design lives on the in-flight `profile-scoped-collection-filters` branch, which has **already implemented** the first cut of the feature but has **not merged to `main` and has not shipped to any client**. Concretely, on this branch today:
|
||||
|
||||
- A migration adds `watch_filter` and `media_filter` as `text` columns on `user_personal_collections`, each with a CHECK constraint (`migrations/sql/20260623184858_user_collection_display_filters.sql`).
|
||||
- `internal/catalog/user_collection_display_filters.go` implements a custom Go filter pass (`FilterUserCollectionDisplayItems`) called from `internal/sections/fetcher.go` and `internal/catalog/catalog_resolver.go`.
|
||||
- Web controls in `web/src/components/collections/CollectionBuilder.tsx`, `web/src/pages/ImportedCollectionEditor.tsx`, and helpers in `web/src/lib/collectionDisplayFilters.ts` already emit the legacy enums.
|
||||
|
||||
Because nothing is merged or released, this is a **clean-replacement** opportunity, not a backward-compatibility problem. The migration can be reshaped before merge; no client depends on the legacy enums. This resolves Open Question #1 below.
|
||||
|
||||
## Problem
|
||||
|
||||
Personal collection display filtering is growing its own filter model alongside two pre-existing completion implementations. There are **three** places that answer "is this item played/read for this profile?":
|
||||
|
||||
- the custom watched/media pass in `internal/catalog/user_collection_display_filters.go` (exact user-collection reads),
|
||||
- the personalized `watched` query rule in `internal/catalog/query_builder.go` (`userStateCompletionClause`),
|
||||
- the item user-data path in `internal/api/handlers/user_state.go` (`allEpisodesCompleted`), which is the canonical played/read state surfaced to clients.
|
||||
|
||||
That creates more than one way to answer the same product question: "should this item be visible for this profile based on watched state?" If multiple paths persist, they will drift in semantics, performance, and bug fixes. The drift is already visible *and already duplicated in code*:
|
||||
|
||||
- The custom personal collection helper treats series/seasons as watched only when all child episodes are complete; `userCollectionAllEpisodesCompleted` is a near-verbatim copy of `allEpisodesCompleted` in `user_state.go`.
|
||||
- The catalog `watched` rule, by contrast, checks completion for the row's own `content_id` only (`uwp.media_item_id = mi.content_id`), so a series row is effectively never "watched" because progress/history is keyed to episode IDs, not the series ID.
|
||||
|
||||
Consolidating onto the catalog predicate therefore removes one Go copy outright and requires teaching the catalog predicate the rollup the other two already perform.
|
||||
|
||||
## Goals
|
||||
|
||||
- Use the catalog query system as the canonical representation and executor for watched filters.
|
||||
- Make personal collection display filtering, catalog filtering, and collection-builder UI speak the same `QueryDefinition` / `QueryRule` language.
|
||||
- Preserve exact collection source order when a collection is rendered as a shelf or source-ordered list.
|
||||
- Preserve existing access controls and profile scoping.
|
||||
- Avoid persisting a second watched-filter vocabulary unless needed as a compatibility shim.
|
||||
- Keep the implementation narrow enough to land as a refactor/standardization of an already intended collection-filter feature.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- No new broad smart-collection DSL.
|
||||
- No change to external sync membership. Imported lists still sync exact item membership before display filtering is applied.
|
||||
- No change to collection ownership, sharing, or profile visibility rules.
|
||||
- No client lockstep requirement. Existing clients can ignore new display-filter capabilities until they adopt the collection editor.
|
||||
- No attempt to optimize very large exact collections beyond the current exact-source behavior in this pass.
|
||||
|
||||
## Scope Gate
|
||||
|
||||
`docs/architecture/v1-scope.md` says v1 is not locked and the proposal window is open. This design should be treated as a standardization of the profile-scoped collection filter work already underway, not as a separate feature expansion. If the final implementation exposes a new public capability beyond personal collection display filtering, file a v1 capability proposal before opening a feature PR.
|
||||
|
||||
## Design Principle
|
||||
|
||||
One feature gets one semantic implementation:
|
||||
|
||||
```json
|
||||
{ "field": "watched", "op": "is", "value": true }
|
||||
```
|
||||
|
||||
is the only canonical expression for "show watched items"; `value: false` is the canonical expression for "show unwatched items".
|
||||
|
||||
UI controls may present simpler labels such as "All", "Watched", and "Unwatched", but those controls must translate into catalog query rules before execution. Backend code should not add another watched evaluator for personal collections.
|
||||
|
||||
## Canonical Semantics
|
||||
|
||||
The canonical `watched` filter should match the user-facing played/read state shown by item user data:
|
||||
|
||||
| Item kind | `watched is true` should mean |
|
||||
|---|---|
|
||||
| Movie / episode / video leaf | Completed progress or visible completed history exists for the active profile. |
|
||||
| Series | At least one child episode exists, and every child episode is completed for the active profile. |
|
||||
| Season | At least one child episode exists, and every child episode in the season is completed for the active profile. |
|
||||
| Ebook | Reader progress is at or above `models.EbookFinishedProgressThreshold`. |
|
||||
| Manga | Use the same read-state source the catalog uses for manga/chapters once manga catalog filtering is finalized; do not add a collection-only interpretation. |
|
||||
|
||||
This intentionally aligns with `internal/api/handlers/user_state.go`, where series and season user data are derived by `allEpisodesCompleted`. If this definition is too expensive for catalog queries, we should optimize the catalog predicate rather than keep a separate personal collection definition.
|
||||
|
||||
For this table to hold when a query is *unscoped* (mixed-media exact collections), the predicate must pick the completion source from each row's own type, not from a single query-wide `media_scope` — see Catalog Watched Predicate Work. Picking the source by `media_scope` alone misclassifies any row whose type differs from the scope (notably ebooks under an unscoped query).
|
||||
|
||||
## Storage Model
|
||||
|
||||
Recommended canonical storage for personal collection display filters:
|
||||
|
||||
- Add a dedicated JSON column named for display filtering, `display_query_definition`. Do **not** overload the existing `query_definition` field (see Risks: it already carries `library_ids` for live/smart collections and is fragile enough to have needed a recent sanitization fix).
|
||||
- Store the standard catalog `QueryDefinition` *shape* in that field, normalized to a filter-only fragment (see normalization rules below).
|
||||
- For the current "media filter" control, store a **`type` rule**, not `media_scope`:
|
||||
- `all` -> no `type` rule
|
||||
- `movie` -> `{ "field": "type", "op": "is", "value": "movie" }`
|
||||
- `series` -> `{ "field": "type", "op": "is", "value": "series" }`
|
||||
- For the current "watch filter" control, store a `watched` rule:
|
||||
- `all` -> no watched rule
|
||||
- `watched` -> `{ "field": "watched", "op": "is", "value": true }`
|
||||
- `unwatched` -> `{ "field": "watched", "op": "is", "value": false }`
|
||||
|
||||
**Why `type` rule, not `media_scope`:** `media_scope` is a single string on `QueryDefinition`, not a rule, and there is no logic to intersect two scopes (only `intersectCatalogDefinitionLibraries` exists, for library IDs). A `type` rule composes as an ordinary AND group and avoids any collision with a smart collection's own `media_scope`. `type` is already a supported, executable query field, so this needs no new merge machinery. Reserve `media_scope` for the source query's own scoping.
|
||||
|
||||
**`display_query_definition` is a filter-only fragment, not a full query.** A full `QueryDefinition` carries fields that are meaningless — and actively dangerous — for a display overlay. In particular the executor honors `def.Limit` internally, so a persisted display query with `limit` set would cap the match set *before* the exact-collection source-order reorder, silently and permanently hiding matching items and producing wrong totals/pages. `sort` and `library_ids` are likewise the source/collection's concern, not the overlay's. Normalize on the API boundary before persisting and before executing:
|
||||
|
||||
- Persist only `match` and `groups` (with rules drawn from the allowed display vocabulary — initially `type` and `watched`).
|
||||
- Reject or strip `limit`, `sort`, `library_ids`, and `media_scope` from the fragment. Prefer reject-with-error on write so clients learn the contract; strip defensively on read for forward-compat.
|
||||
- Storing the `QueryDefinition` *shape* keeps the door open to a richer rule set later; it does not mean accepting the whole `QueryDefinition` surface.
|
||||
|
||||
Compatibility: none required. The legacy `watch_filter` / `media_filter` enums exist only on this unmerged branch (see Current Branch State), so they are dropped rather than aliased. If — and only if — a future build ships the enums to clients before this lands, revisit the compatibility shim; until then there is no legacy surface to preserve.
|
||||
|
||||
## Execution Model
|
||||
|
||||
### Smart/live collections
|
||||
|
||||
Smart and live collections already resolve through `CatalogResolver.resolveQuerySource`. Their stored query is already a catalog `QueryDefinition`. Display filters should be merged into that query before execution.
|
||||
|
||||
**Merge must guarantee `(source) AND (display)` — appending a group does not.** `QueryBuilder.Build` joins *all* top-level groups with a single joiner derived from the definition's top-level `match` (`query_builder.go`: `topJoiner` is ` OR ` when `match == "any"`). So a smart/live source with `match: any` and groups `G1, G2` becomes `G1 OR G2`; appending a display group `D` yields `G1 OR G2 OR D` — which *widens* the result past the collection source and lets contradictory watched rules return rows instead of none. The QueryDefinition model has only one top-level `match` and only two levels (groups of rules), so there is no in-model way to express `(G1 OR G2) AND D`.
|
||||
|
||||
Merge rule:
|
||||
|
||||
- Keep library scope, `media_scope`, limit, and sort from the collection's source query.
|
||||
- Combine the source definition and the display fragment with a true conjunction that is independent of either side's `match`. The robust implementation is an **executor-level conjunction**: build the source predicate and the display predicate as two separate SQL fragments and join them with ` AND ` (each side already parenthesized), rather than merging group lists. This is net-new merge logic, not an existing capability.
|
||||
- The append-as-group shortcut is correct **only** when the source's top-level `match == "all"`. When `match == "any"` it is incorrect. Do not special-case "all" and forget "any"; implement the general conjunction and test both.
|
||||
- Because the media control is stored as a `type` rule (not `media_scope`), it ANDs in through the same conjunction. If the source query's `media_scope` already excludes that type (e.g. a movies-only smart collection plus a `type is series` display filter), the result is empty — no scope-intersection code is needed. (There is no existing media-scope intersection logic to lean on, which is the main reason the media control is modeled as a `type` rule rather than a second `media_scope`.)
|
||||
- If both the source and display layers contain `watched` rules, the conjunction preserves both; contradictory rules naturally return no rows.
|
||||
|
||||
### Exact/manual/imported collections
|
||||
|
||||
Exact collections have a source item set from `user_personal_collection_items`. Display filtering is applied after membership is known but before pagination.
|
||||
|
||||
**This path replaces two existing filters, not one.** Today the source-ordered exact path runs the in-Go `filterCatalogItems` (a metadata-only matcher that cannot evaluate personalized fields — it returns `false` for `watched`) *and* the separate `FilterUserCollectionDisplayItems` Go helper for the watched dimension. Neither is the SQL executor. Routing through the executor folds both metadata and personalized filtering into one SQL path; `FilterUserCollectionDisplayItems` is deleted and `filterCatalogItems` is no longer used for these collections.
|
||||
|
||||
Recommended execution:
|
||||
|
||||
1. Load collection item IDs in source order (from `user_personal_collection_items`, `ORDER BY position`).
|
||||
2. Build a catalog `QueryDefinition` from the normalized display fragment plus any request overlay. Force `Limit = nil` and drop any fragment-level `sort`/`library_ids` so the executor cannot truncate or reorder the match set out from under step 4. (Defense in depth: the fragment is already normalized at persistence per the Storage Model, but the executor call must not reintroduce a limit.)
|
||||
3. Run the normal catalog executor with `AccessFilter.AllowedContentIDs = itemIDs` so all filtering, including `watched`, uses the catalog query path. (`resolveCandidateItemsWithQuery` already establishes this pattern.)
|
||||
4. For source-ordered requests, retrieve the *full* matching set (no limit), collect the matching IDs into a set, reapply the original source order, and only then apply request offset/limit. Request pagination is applied after reordering, never inside the executor.
|
||||
5. For requests that sort via the query, allow the catalog executor to sort normally. **Note:** exact user collections are hardcoded to source order today (`UseSourceOrder: true`); this branch exists for forward-compatibility and is not exercised by current callers. Mark it clearly so it is not mistaken for present behavior.
|
||||
|
||||
This preserves exact collection order without duplicating watched logic. Step 4 is a new shape (run the executor for the match-set, then reorder), distinct from the sorted-pagination shape `resolveCandidateItemsWithQuery` uses today — call out the differing limit handling in the implementation.
|
||||
|
||||
## Catalog Watched Predicate Work
|
||||
|
||||
The existing catalog `watched` rule is already implemented but needs one semantic hardening before it becomes the shared path:
|
||||
|
||||
- Extract a reusable played/read predicate for `QueryBuilder`.
|
||||
- Keep the existing leaf-item checks for progress, completed history, hidden history, and ebook reader progress.
|
||||
- Add series and season rollup branches so `watched` matches item user-data semantics.
|
||||
- Add SQL tests that assert series/season predicates are present only when relevant.
|
||||
- Reuse the same predicate for `last_watched` only if the semantics truly match; otherwise leave `last_watched` as a leaf/history timestamp filter and document the difference. (`last_watched` is a timestamp aggregate built from the `user_last_watched` CTE — different machinery from the completion predicate — so it most likely stays separate.)
|
||||
|
||||
**Episode-set parity is the load-bearing requirement, not the rollup branch itself.** The Go rollups (`allEpisodesCompleted` in `user_state.go` and its copy in the collection helper) define "all episodes complete" over the episode set returned by the episode repository (`ListBySeriesIDs` / `ListBySeasonIDs`). If the SQL rollup's notion of "child episodes" differs — specials, unaired, or unavailable episodes included or excluded differently — the catalog `watched` rule will *disagree* with item user-data, recreating the exact drift this design removes, just relocated into SQL. The hardening must therefore reproduce the repository's child-episode set exactly (same joins, same filters), and a test must pin that equivalence (see Testing Strategy). Treat Open Question #4 (all-available vs all-aired) as a question about *what that repository set already is*, and match it.
|
||||
|
||||
**Row-type branching is mandatory for unscoped queries, including ebooks.** The current predicate selects the ebook source (`ebook_reader_progress`) *only* when `mediaScope == "ebook"`; otherwise it uses video watch progress/history (`userStateCompletionClause` branches on `qb.mediaScope == "ebook"`). Exact collection membership is not type-restricted, so an exact collection can contain an ebook while the query runs unscoped (the `all` media control emits no `type` rule). Under today's code that ebook would be evaluated against video progress/history — i.e. classified as never-watched regardless of read state — reintroducing exactly the kind of drift this design removes. The hardened predicate must therefore branch by the row's own type when the query is unscoped:
|
||||
|
||||
- movie / episode / video leaf -> watch progress + completed history,
|
||||
- series / season -> episode rollup (with repository-parity episode sets, above),
|
||||
- ebook -> `ebook_reader_progress >= EbookFinishedProgressThreshold`,
|
||||
- manga -> deferred (see Non-Goals / Canonical Semantics).
|
||||
|
||||
If row-type branching cannot land in the first cut, v1 must instead *enforce* a video-only scope at execution for exact collections (so ebooks are never silently classified by the video source) and say so explicitly — but row-type branching is the correct end state because it is what makes the canonical semantics table hold under unscoped queries.
|
||||
|
||||
The first implementation may still gate series/season rollup behavior by `mediaScope == "series"` and exact known season rows, but it must not assume a single item kind for unscoped/video queries that can return movies, series, and ebooks together.
|
||||
|
||||
## API Shape
|
||||
|
||||
Preferred API model:
|
||||
|
||||
- Add `display_query_definition?: QueryDefinition` to personal collection create/update/response types.
|
||||
- Replace the branch's `watch_filter` / `media_filter` request and response fields with `display_query_definition`. Since the enums are unreleased (see Current Branch State), no aliasing or deprecation header flow is required.
|
||||
- Keep collection capabilities additive:
|
||||
- expose supported display-query fields using the existing catalog field vocabulary,
|
||||
- optionally expose UI presets for "all/watched/unwatched" and "all/movies/series" as client convenience.
|
||||
|
||||
Do not add a separate `filter_watched` query parameter to catalog endpoints. Catalog already has the watched rule.
|
||||
|
||||
## UI Model
|
||||
|
||||
The web UI can keep simple controls:
|
||||
|
||||
- media segmented control: All / Movies / Series,
|
||||
- watched segmented control: All / Unwatched / Watched.
|
||||
|
||||
Those controls should edit a `QueryDefinition` fragment using shared helpers, not a separate collection-only state model.
|
||||
|
||||
Recommended helper location:
|
||||
|
||||
- `web/src/lib/collectionDisplayFilters.ts` should convert between UI presets and catalog `QueryDefinition`.
|
||||
- `web/src/components/collections/CollectionGuidedRulesEditor.tsx` remains the broader catalog-query editor.
|
||||
- `web/src/components/collections/CollectionBuilder.tsx` and `web/src/pages/ImportedCollectionEditor.tsx` use the shared helper.
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
The legacy `watch_filter` / `media_filter` enums exist only on this unmerged branch and have shipped to no client (see Current Branch State), so the clean-replacement path applies. There is no backfill, dual-read, or deprecation window:
|
||||
|
||||
- Reshape the existing migration `20260623184858_user_collection_display_filters.sql` rather than layering a second migration on top: drop the `watch_filter` / `media_filter` columns and their CHECK constraints, and add the canonical `display_query_definition jsonb` column (nullable; `NULL` / absent means "no display filter"). Because the branch is unmerged, editing the not-yet-applied migration in place is cleaner than a forward-then-backfill sequence.
|
||||
- Update API and web types to use display-query-backed helpers; delete the legacy enum request/response fields.
|
||||
- No server code reads or writes the legacy columns after this change.
|
||||
|
||||
A backfill/dual-read migration would only be warranted if a build ships the legacy enums to clients before this lands. If that happens, fall back to: add `display_query_definition`, backfill (`watch_filter = 'watched'` -> watched rule true; `'unwatched'` -> false; `media_filter = 'movie'`/`'series'` -> `type` rule; `all` omits the rule), read legacy as fallback, stop writing legacy, and remove later under the API deprecation flow. Treat this as a contingency, not the plan of record.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Merge OR-leak (smart/live):** appending a display group to a `match: any` source query widens results past the collection source instead of constraining them. Mitigate with the executor-level conjunction (Execution Model) and a test that a `match: any` source plus a display filter never returns rows the source alone would exclude.
|
||||
- **Display fragment truncation:** a persisted `limit` (or stray `sort`) on `display_query_definition` would cap or reorder the exact-collection match set before the source-order reorder, hiding items and corrupting totals. Mitigate by normalizing the fragment to filter-only on write and forcing `Limit = nil` on the match-set execution.
|
||||
- **Ebook misclassification under unscoped queries:** an ebook in a mixed-media exact collection evaluated by an unscoped `watched` rule is classified by the video source today (`media_scope`-gated ebook predicate). Mitigate with row-type branching in the predicate, or enforce a video-only execution scope for v1 (Catalog Watched Predicate Work).
|
||||
- **Episode-set divergence (highest risk):** if the SQL rollup's child-episode set differs from the repository's `ListBySeriesIDs` / `ListBySeasonIDs` listing, the catalog `watched` rule will disagree with item user-data and reintroduce drift in a new place. Mitigate by reproducing the repository's joins/filters exactly and pinning the equivalence with a test. This, not raw SQL cost, is the thing most likely to ship subtly wrong.
|
||||
- **Series/season SQL cost:** all-episodes-complete checks are more expensive than leaf checks. Mitigate by only emitting rollup branches when item types can include series/seasons and by relying on indexes on episode parent IDs and progress/history media IDs.
|
||||
- **Semantics change for catalog series filters:** extending catalog `watched` to match displayed played state can change existing catalog results for series (today they are effectively never "watched"). This is likely correct, but it must be called out in release notes or PR text.
|
||||
- **Exact collection performance:** source-order rendering requires evaluating the full exact collection before pagination (the executor must return the complete match-set to reorder). This matches current behavior but should not regress further; very large exact collections remain out of scope this pass (see Non-Goals).
|
||||
- **Field overloading (resolved → dedicated column):** code inspection confirms `query_definition` already carries `library_ids` for live/smart collections and recently needed a `library_ids` sanitization fix. Reusing it for display filters would entangle two concerns; use a dedicated `display_query_definition` column.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Backend tests are justified because this changes shared personalized filtering semantics:
|
||||
|
||||
- **Equivalence test (highest value):** for the same series/season/profile fixtures, assert the catalog `watched` rule and `user_state.go`'s `Played` flag agree. This is the "one semantic" goal made executable and is the test most likely to catch episode-set divergence; it should fail loudly if the two definitions drift.
|
||||
- `internal/catalog/query_builder_test.go`: `watched` emits leaf completion checks and series/season rollup checks as appropriate.
|
||||
- `internal/catalog/query_executor_test.go` or focused resolver tests: exact collection filtering routes through catalog `watched` and preserves source order; the executor returns the full match-set so reordering is stable across pages.
|
||||
- `internal/catalog/catalog_resolver_test.go`: contradictory display/source filters return no rows; a `type is series` display filter on a movies-only smart collection yields no rows without any scope-intersection code.
|
||||
- **Merge safety:** a smart/live source with top-level `match: any` plus a display filter must never return rows the source alone excludes (guards against the OR-leak); a contradictory `watched` overlay on a `match: any` source returns zero rows.
|
||||
- **Fragment normalization:** a `display_query_definition` carrying `limit`/`sort`/`library_ids` is rejected on write (or stripped on read); an exact collection larger than any plausible page returns the full match set in source order regardless of fragment contents.
|
||||
- **Ebook under unscoped watched:** a mixed-media exact collection (movie + ebook) with an `unwatched`/`watched` filter classifies the ebook via `ebook_reader_progress`, not video history (guards against the misclassification finding).
|
||||
- `internal/api/handlers/collections_test.go`: create/update round-trips `display_query_definition`; the removed `watch_filter` / `media_filter` fields are gone from request and response.
|
||||
- Migration test only if the contingency backfill path is used (it is not, on the plan of record).
|
||||
|
||||
Frontend tests should stay targeted:
|
||||
|
||||
- helper tests for converting simple UI controls to/from `QueryDefinition`,
|
||||
- one editor test proving the submit payload carries the canonical display query.
|
||||
|
||||
## Implementation Outline
|
||||
|
||||
1. Harden catalog `watched` semantics (series/season rollup with repository-parity episode sets) and land the equivalence test. This is independently valuable and reviewable; consider shipping it as its own PR first.
|
||||
2. Reshape migration `20260623184858` to drop `watch_filter` / `media_filter` and add `display_query_definition jsonb`.
|
||||
3. Add display-query types and normalization helpers in backend collection handling.
|
||||
4. Replace both the custom `FilterUserCollectionDisplayItems` pass and the in-Go `filterCatalogItems` source-order path with catalog-query execution constrained by exact item IDs (match-set, then reorder).
|
||||
5. Update web controls to emit catalog query fragments through shared helpers; delete the legacy enum fields.
|
||||
6. Run focused Go tests for catalog/query/user-collection paths and targeted frontend helper tests.
|
||||
|
||||
Per the v1 process, link the capability sub-issue (`Part of #NNN`) and keep PRs to one concern — predicate-hardening (step 1) is a natural standalone PR separate from the storage/UI reshape (steps 2–5).
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Resolved.** The legacy enums exist only on this unmerged, unreleased branch, so it replaces them cleanly with no compatibility shim (see Current Branch State and Migration Strategy).
|
||||
- **Decided:** the standardized *controls* expose video item scopes (movie/series/episode) for v1; manga read-state filtering is deferred until catalog manga filtering is finalized, then adopts the same catalog read-state source rather than a collection-only interpretation. Note this constrains the *control vocabulary*, not collection *membership* — exact collections can still contain ebooks, so the watched predicate must branch by row type (see Catalog Watched Predicate Work) regardless of what the control offers.
|
||||
- **Decided:** first version exposes only the `type` and `watched` controls, stored as a **filter-only `QueryDefinition` fragment** (`match` + `groups`; no `limit`/`sort`/`library_ids`/`media_scope`). Keeping the fragment shape lets a richer rule set turn on later without a storage change, while normalization prevents the fragment from carrying execution-controlling fields (see Storage Model).
|
||||
- Should catalog `watched` rollup for series mean "all available episodes complete" or "all aired episodes complete"? This is a question about *what the episode repository already returns* — match that set exactly (see Catalog Watched Predicate Work / episode-set parity). Only change the definition by changing the repository path and item user-data together, never the catalog rule alone.
|
||||
- **New:** confirm no `silo-android` / `silo-apple` build has already consumed the branch's `watch_filter` / `media_filter` fields. The design assumes not (the branch is unreleased); if a client has pinned them, the contingency backfill path applies instead.
|
||||
@@ -0,0 +1,220 @@
|
||||
# Download limits & restrictions (downloads v2) — design
|
||||
|
||||
**Goal:** Extend downloads v2 with the restriction knobs admins actually need: (1) a **quality
|
||||
ceiling** — server-wide and per-user — so a user can be allowed to download, even transcode, but
|
||||
only up to a capped bitrate; (2) a **batch size cap** protecting the server from a single series
|
||||
request fanning out into hundreds of rows and transcode jobs; (3) **per-user overrides** for the
|
||||
existing global quantity and bandwidth limits, following the established `users.max_streams` /
|
||||
`users.max_transcodes` pattern, so households can tier users without changing the global policy.
|
||||
|
||||
**Status:** Draft — design input for a v1 capability proposal (Silo v1 scope gate,
|
||||
`docs/architecture/v1-scope.md`). All API changes are additive within `/api/v1`.
|
||||
|
||||
Decisions locked with the requester:
|
||||
- **No device/user storage caps.** Device disk is the client's job (per-subscription
|
||||
`max_storage_bytes` already covers the monitoring case); server disk is already covered by
|
||||
`download.artifact_max_bytes`; serving volume is already covered by bandwidth + per-period quotas.
|
||||
- **No expiry / lease / revocation of already-downloaded files.** Unchanged from the downloads v2
|
||||
design (`docs/superpowers/specs/2026-06-18-offline-sync-mobile-design.md`).
|
||||
- **Transcode restriction already exists** (`download.transcode_enabled` +
|
||||
`users.download_transcode_allowed`) and is kept as-is; the quality ceiling composes with it
|
||||
("transcode allowed, but only up to 5mbps").
|
||||
- **Admin observability (downloads dashboard, admin list/revoke endpoints) is a separate effort**,
|
||||
not part of this spec.
|
||||
|
||||
> Commands and paths in this document are repository-relative; assume the repository root is the cwd.
|
||||
|
||||
## Current state (what this builds on)
|
||||
|
||||
Enforcement today lives in `internal/downloads`:
|
||||
|
||||
- **Policy** — `policy.go`: `DownloadQualityResolver.Resolve` validates a requested quality from the
|
||||
ladder `original > 20mbps > 10mbps > 5mbps > 2mbps > 1mbps` and gates bitrate presets on
|
||||
`cfg.TranscodeEnabled && user.DownloadTranscodeAllowed`. `PresetsFor` advertises the fulfillable
|
||||
ladder through `GET /api/v1/downloads/capability` (`quality_presets`), so clients render only what
|
||||
the server will honor.
|
||||
- **Quantity** — `limiter.go`: `QuantityLimiter.Check(ctx, userID, batchSize)` enforces the global
|
||||
`download.max_concurrent_per_user` and `download.max_per_period` / `download.period_duration`
|
||||
settings (0 = unlimited) against `CountActiveByUser` / `CountByUserSince`.
|
||||
- **Bandwidth** — `bandwidth.go`: `BandwidthManager` token-buckets file serving with one server-wide
|
||||
rate and one per-user rate (`download.server_bandwidth_mbps`, `download.user_bandwidth_mbps`),
|
||||
caching one `rate.Limiter` per user id.
|
||||
- **Per-user flags** — `users.download_allowed`, `users.download_transcode_allowed`
|
||||
(`internal/auth/repository.go`), exposed on the admin users API (`internal/api/handlers/admin.go`)
|
||||
and web UI (`web/src/pages/AdminUserDetail.tsx`).
|
||||
- **Bulk** — `service.go` `createSeriesScoped` fans a series/season request into managed entries with
|
||||
no upper bound; `resolveBulkQuality` intentionally keeps batches **original-only**
|
||||
(`ErrBulkQualityUnavailable`). Subscription sync auto-registers in-scope episodes without
|
||||
consuming the quantity limiter (by design).
|
||||
- **Serve-time re-check** — `Service.ServeFile` re-resolves policy on every file request, so
|
||||
disabling downloads or revoking a user flag takes effect immediately for anything not yet pulled.
|
||||
|
||||
Settings are freeform keys with defaults and validation in `internal/config/db_loader.go`, hot-
|
||||
reloaded by `Service.loadConfig` (which also reloads the limiter and bandwidth manager on change),
|
||||
and edited in `web/src/pages/admin-settings/DownloadSettings.tsx`.
|
||||
|
||||
## 1. Quality ceiling
|
||||
|
||||
### Semantics
|
||||
|
||||
- New server setting **`download.max_quality`** (string; `""` = no cap; otherwise one of the quality
|
||||
ladder values). New nullable per-user column **`users.download_max_quality`** (`NULL` = inherit the
|
||||
server setting; otherwise one of the ladder values).
|
||||
- The **effective ceiling** for a user is the *lower* of the server setting and the user column on
|
||||
the ordered ladder (`original` is the top). A ceiling below `original` deliberately forbids
|
||||
`original` — that is the point: "this user gets at most a 5mbps transcode, never the full file."
|
||||
- Enforcement is **presets-first**: `PresetsFor` truncates the advertised ladder at the effective
|
||||
ceiling, so compliant clients never offer a forbidden quality and **no client changes are
|
||||
required** (clients already read `quality_presets` from the capability endpoint).
|
||||
- `Resolve` independently rejects any request above the ceiling with a new sentinel
|
||||
`ErrQualityNotPermitted` (handler maps it alongside the existing limit errors as HTTP 403,
|
||||
distinct from `ErrInvalidQuality`), so a non-compliant client cannot bypass the cap.
|
||||
- The **original-compatibility fallback** inside `Resolve` (original → remux/transcode when the
|
||||
device can't play the source) must clamp its chosen `EffectiveQuality`/bitrate to the ceiling.
|
||||
In practice a sub-original ceiling makes the fallback moot — `original` is already rejected — but
|
||||
the clamp keeps the invariant local to `Resolve` rather than depending on call-site ordering.
|
||||
- **Serve-time**: the existing `ServeFile` policy re-check additionally denies rows whose
|
||||
`effective_quality` exceeds the user's *current* ceiling. Consistent with today's revocation
|
||||
semantics: lowering a cap stops future serving of higher-quality registered entries; files already
|
||||
on-device are untouched (no expiry).
|
||||
|
||||
### Interactions (documented behavior, not bugs)
|
||||
|
||||
- **Ceiling below `original` + `download_transcode_allowed = false`** → the fulfillable preset list
|
||||
is empty and the capability endpoint reports no presets; the user effectively cannot download.
|
||||
This is legal admin configuration. The admin UI shows a hint on the per-user field when this
|
||||
combination is selected ("a sub-original cap requires the transcode permission").
|
||||
- **Series/season/monitoring downloads are original-only today** (`resolveBulkQuality`). A user
|
||||
capped below `original` therefore cannot create series/season batches or receive subscription
|
||||
auto-registrations; those paths return `ErrQualityNotPermitted` (create) or skip registration with
|
||||
a `SkippedDownload` reason `quality_not_permitted` (subscription sync). This is accepted for now
|
||||
and called out here so it is not rediscovered as a bug. Lifting it means bulk transcode — a
|
||||
deliberate follow-up, not smuggled into this change.
|
||||
|
||||
### Implementation
|
||||
|
||||
- `Resolve`/`PresetsFor` gain the ceiling input. Rather than growing the parameter lists further,
|
||||
introduce a small `PolicyInput`/`userPolicy` struct (user + cfg + artifactsAvailable) if the
|
||||
signatures get unwieldy — implementer's choice; behavior is what is specified above.
|
||||
- Ladder comparison helper in `internal/downloads/model.go` (e.g. `QualityAtMost(q, ceiling) bool`
|
||||
using the `QualityPresets` order) — no string parsing of bitrates.
|
||||
- Validation: `db_loader.go` rejects a `download.max_quality` value not on the ladder; the users
|
||||
API rejects invalid `download_max_quality` the same way (shared validation via
|
||||
`downloads.ValidQuality`, with `""`/`NULL` allowed).
|
||||
|
||||
## 2. Batch size cap
|
||||
|
||||
- New server setting **`download.max_batch_size`** (int; default **100**; 0 = unlimited): the
|
||||
maximum number of managed entries one explicit series/season create may register, and the maximum
|
||||
auto-registered per subscription-sync pass.
|
||||
- **Explicit series/season create** (`createSeriesScoped`): if the in-scope episode count (after
|
||||
access filtering and skip computation) exceeds the cap, **reject** the whole request with a new
|
||||
sentinel `ErrBatchTooLarge` (HTTP 400, error body includes the cap so clients can message "try a
|
||||
season at a time or subscribe"). Rejection is chosen over truncation because a silently partial
|
||||
one-shot batch never self-heals — there is no later pass that picks up the remainder.
|
||||
- **Subscription sync** (`subscription_service.go`): **truncate** each pass at the cap (stable
|
||||
episode order). Truncation is correct here because sync recomputes in-scope-but-unregistered
|
||||
episodes every pass, so the remainder registers on subsequent syncs — natural pagination, and one
|
||||
monster series cannot monopolize a sync pass or the artifact queue.
|
||||
- The capability response additively gains **`max_batch_size`** (int, 0 = unlimited) so clients can
|
||||
preflight ("this series has 340 episodes; the server accepts 100 per request") instead of
|
||||
discovering the limit by error.
|
||||
|
||||
## 3. Per-user quantity & bandwidth overrides
|
||||
|
||||
- Three new nullable columns on `users`, mirroring the `max_streams` pattern
|
||||
(`internal/models/user.go`, `internal/auth/repository.go`):
|
||||
- `download_max_concurrent int` — overrides `download.max_concurrent_per_user`
|
||||
- `download_max_per_period int` — overrides `download.max_per_period` (window stays the global
|
||||
`download.period_duration`; a per-user window is deliberately out of scope)
|
||||
- `download_bandwidth_mbps int` — overrides `download.user_bandwidth_mbps`
|
||||
- Semantics: `NULL` = inherit the global setting; **0 = explicitly unlimited** (matches the global
|
||||
settings' zero-means-unlimited convention, and is the reason `NULL` and `0` must stay distinct).
|
||||
- **QuantityLimiter**: `Check` gains the caller-resolved user overrides — e.g.
|
||||
`Check(ctx, userID, batchSize, overrides Overrides)` where `Overrides` carries the two optional
|
||||
ints. Every existing call site already has the `*models.User` loaded (the create paths load it
|
||||
for policy), so resolution stays in `service.go` and the limiter remains storage-free.
|
||||
- **BandwidthManager**: the per-user limiter cache currently assumes one global rate. Store the
|
||||
rate alongside each cached limiter and recreate the entry when the effective rate for that user
|
||||
changes (`ThrottledReader` gains the resolved per-user bps from the caller; `ServeFile` already
|
||||
loads the user for its policy re-check). `Reload` keeps its existing clear-all behavior.
|
||||
- **Admin API**: the three fields (plus `download_max_quality` from §1) are added to the admin user
|
||||
create/update/response DTOs in `internal/api/handlers/admin.go` and to
|
||||
`UpdateUserInput`/`CreateUserInput` in `internal/models/user.go` +
|
||||
`internal/auth/repository.go` — the exact `max_streams` additive pattern.
|
||||
|
||||
## Data model (Goose migration)
|
||||
|
||||
One timestamped migration (`make migrate-create NAME=user_download_limits`):
|
||||
|
||||
```sql
|
||||
ALTER TABLE users
|
||||
ADD COLUMN download_max_quality text,
|
||||
ADD COLUMN download_max_concurrent integer,
|
||||
ADD COLUMN download_max_per_period integer,
|
||||
ADD COLUMN download_bandwidth_mbps integer;
|
||||
```
|
||||
|
||||
All nullable, no defaults, no backfill — `NULL` means "inherit global" for every column. The two new
|
||||
settings keys (`download.max_quality`, `download.max_batch_size`) are settings rows, not schema.
|
||||
|
||||
## API surface (all additive)
|
||||
|
||||
- `GET /api/v1/downloads/capability` — `quality_presets` now reflects the effective ceiling
|
||||
(existing field, narrowed values); new field `max_batch_size`.
|
||||
- `POST /api/v1/downloads` — new 403 error case (`quality_not_permitted`) and new 400 error case
|
||||
(`batch_too_large`, body includes the configured cap).
|
||||
- `POST /api/v1/downloads/subscriptions/sync` — skipped registrations may carry the new
|
||||
`SkippedDownload` reason `quality_not_permitted`.
|
||||
- Admin users endpoints — four new optional request fields / response fields (§1, §3).
|
||||
- Settings endpoints — two new keys; both hot-reloadable (no restart-required entry in
|
||||
`internal/config/restart_keys.go`).
|
||||
|
||||
## Web admin UI
|
||||
|
||||
- `web/src/pages/admin-settings/DownloadSettings.tsx`: "Max Download Quality" (select over the
|
||||
ladder + "No limit") under Quantity Limits or a new Restrictions group; "Max Batch Size" numeric
|
||||
field with the 0-=-unlimited hint.
|
||||
- `web/src/pages/AdminUserDetail.tsx` (+ the user edit dialog): rows for the four per-user fields,
|
||||
each showing the inherited global value when unset (as the streams/transcodes fields do), plus the
|
||||
§1 hint when a sub-original cap is combined with transcode-not-allowed.
|
||||
|
||||
## Enforcement summary
|
||||
|
||||
| Restriction | Hard/soft | Enforced at |
|
||||
|---|---|---|
|
||||
| Quality ceiling | Hard | `Resolve` (create), `PresetsFor` (capability), `ServeFile` re-check |
|
||||
| Batch size cap | Hard | `createSeriesScoped` (reject), subscription sync (truncate per pass) |
|
||||
| Concurrent / per-period overrides | Hard | `QuantityLimiter.Check` on every create path |
|
||||
| Bandwidth override | Hard | `ThrottledReader` on every byte served |
|
||||
|
||||
Nothing in this spec is client-enforced; the client-pull architecture is unchanged.
|
||||
|
||||
## Client follow-up (silo-android, silo-apple)
|
||||
|
||||
- **Quality ceiling / presets:** zero work if the client renders `quality_presets` from the
|
||||
capability endpoint (the downloads v2 contract). Audit for hardcoded ladders.
|
||||
- **Batch errors:** handle `batch_too_large` with a useful prompt (download per season / subscribe
|
||||
instead of a raw error toast). Handle empty `quality_presets` as "downloads unavailable for this
|
||||
account", which the capability contract already implies.
|
||||
- No manifest or sync-protocol changes.
|
||||
|
||||
## Testing
|
||||
|
||||
- `policy_test.go`: ceiling truncation of `PresetsFor` (server-only, user-only, both, ties at
|
||||
`original`); `Resolve` rejection above the ceiling; fallback clamp; the
|
||||
cap-without-transcode-permission → empty presets case.
|
||||
- `limiter_test.go` (new cases): override precedence NULL/0/positive vs. global for both limits.
|
||||
- `service_*_test.go`: batch reject at cap boundary (cap, cap+1); subscription sync truncation and
|
||||
next-pass continuation; serve-time denial after a cap is lowered.
|
||||
- Handler tests for the new error mappings and admin-user field round-trips — run in a
|
||||
libvips-capable container (bare-host `go test ./...` skips `internal/api/handlers`).
|
||||
|
||||
## Out of scope / follow-ups
|
||||
|
||||
- Admin downloads dashboard + admin list/revoke endpoints (observability; separate spec).
|
||||
- Bulk (series/season) transcode quality — prerequisite for capped users to use series downloads
|
||||
and monitoring; lifts the `resolveBulkQuality` original-only rule.
|
||||
- Profile-level download permission (parental-controls integration; downloads are already keyed by
|
||||
`profile_id`, so the data model is ready).
|
||||
- Per-user managed-device count cap — revisit only if storage/quantity limits prove insufficient.
|
||||
@@ -31,6 +31,7 @@ require (
|
||||
github.com/oklog/ulid/v2 v2.1.0
|
||||
github.com/pgvector/pgvector-go v0.3.0
|
||||
github.com/pressly/goose/v3 v3.27.1
|
||||
github.com/tetratelabs/wazero v1.12.0
|
||||
github.com/wneessen/go-mail v0.7.3
|
||||
github.com/zishang520/socket.io/v2 v2.5.0
|
||||
go.n16f.net/thumbhash v1.1.0
|
||||
@@ -76,7 +77,7 @@ require (
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.6.0
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.8.1
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
|
||||
|
||||
@@ -4,8 +4,8 @@ github.com/PuerkitoBio/goquery v1.8.0 h1:PJTF7AmFCFKk1N6V6jmKfrNH9tV5pNE6lZMkG0g
|
||||
github.com/PuerkitoBio/goquery v1.8.0/go.mod h1:ypIiRMtY7COPGk+I/YbZLbxsxn9g5ejnI2HSMtkjZvI=
|
||||
github.com/SherClockHolmes/webpush-go v1.4.0 h1:ocnzNKWN23T9nvHi6IfyrQjkIc0oJWv1B1pULsf9i3s=
|
||||
github.com/SherClockHolmes/webpush-go v1.4.0/go.mod h1:XSq8pKX11vNV8MJEMwjrlTkxhAj1zKfxmyhdV7Pd6UA=
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.6.0 h1:Gi9TdH9kt7b8X4xRXH493/nSYb9n0GO4VCWmlll0hKI=
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.6.0/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.8.1 h1:6DLm2s/O3N9KTx2d09Nq3bbV8z4u9HJwO2gueQDPyPw=
|
||||
github.com/Silo-Server/silo-plugin-sdk v0.8.1/go.mod h1:etqmxLTwjxpFH9goAjBDfNDoqHMv2/sqUXu8yx3hNfA=
|
||||
github.com/abadojack/whatlanggo v1.0.1 h1:19N6YogDnf71CTHm3Mp2qhYfkRdyvbgwWdd2EPxJRG4=
|
||||
github.com/abadojack/whatlanggo v1.0.1/go.mod h1:66WiQbSbJBIlOZMsvbKe5m6pzQovxCH9B/K8tQB2uoc=
|
||||
github.com/andybalholm/brotli v1.2.1 h1:R+f5xP285VArJDRgowrfb9DqL18yVK0gKAW/F+eTWro=
|
||||
@@ -197,6 +197,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
|
||||
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU=
|
||||
github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0=
|
||||
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
|
||||
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
|
||||
github.com/uptrace/bun v1.1.12 h1:sOjDVHxNTuM6dNGaba0wUuz7KvDE1BmNu9Gqs2gJSXQ=
|
||||
|
||||
@@ -86,9 +86,15 @@ func (s stubStore) GetProgress(context.Context, string, string) (*userstore.Watc
|
||||
func (s stubStore) ListProgress(context.Context, string, string, int, int) ([]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressFiltered(context.Context, string, string, []string, *int, int, int) ([]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressByMediaItems(context.Context, string, []string) (map[string]userstore.WatchProgress, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListProgressSince(context.Context, string, string) ([]userstore.WatchProgress, string, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) AddHistory(context.Context, userstore.WatchHistoryEntry) error { panic("unused") }
|
||||
func (s stubStore) AddHistoryIfMissing(context.Context, userstore.WatchHistoryEntry) (bool, error) {
|
||||
panic("unused")
|
||||
@@ -99,6 +105,9 @@ func (s stubStore) ListHistory(context.Context, string, int, int) ([]userstore.W
|
||||
func (s stubStore) ListCompletedHistory(context.Context, userstore.CompletedHistoryQuery) ([]userstore.WatchHistoryEntry, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListCompletedHistoryItems(context.Context, userstore.CompletedHistoryItemQuery) ([]userstore.CompletedHistoryItem, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) RemoveHistoryItems(context.Context, string, []string, time.Time) error {
|
||||
panic("unused")
|
||||
}
|
||||
@@ -127,9 +136,18 @@ func (s stubStore) ListFavorites(context.Context, string, int, int) ([]userstore
|
||||
func (s stubStore) ListFavoritesByMediaItems(context.Context, string, []string) (map[string]bool, error) {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
|
||||
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) IsFavorite(context.Context, string, string) (bool, error) { panic("unused") }
|
||||
func (s stubStore) AddToWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) AddToWatchlistAt(context.Context, string, string, time.Time) error {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) RemoveWatchedFromWatchlist(context.Context, string) (bool, error) {
|
||||
return true, nil
|
||||
}
|
||||
func (s stubStore) RemoveFromWatchlist(context.Context, string, string) error { panic("unused") }
|
||||
func (s stubStore) ReplaceWatchlistOrder(context.Context, string, []string) error {
|
||||
panic("unused")
|
||||
}
|
||||
func (s stubStore) ListWatchlist(context.Context, string, int, int) ([]userstore.WatchlistEntry, error) {
|
||||
panic("unused")
|
||||
}
|
||||
|
||||
@@ -89,6 +89,7 @@ func (l *PGLibraryRefreshItemLister) ListLibraryItems(ctx context.Context, libra
|
||||
query := `
|
||||
SELECT mi.content_id, COALESCE(mi.tmdb_id, ''), COALESCE(mi.tvdb_id, ''), COALESCE(mi.imdb_id, '')
|
||||
FROM media_item_libraries mil
|
||||
JOIN media_folders f ON f.id = mil.media_folder_id
|
||||
JOIN media_items mi ON mi.content_id = mil.content_id
|
||||
WHERE mil.media_folder_id = $1`
|
||||
args := []any{libraryID}
|
||||
@@ -123,6 +124,11 @@ func (l *PGLibraryRefreshItemLister) ListLibraryItems(ctx context.Context, libra
|
||||
FROM stale_media_ids smi
|
||||
WHERE smi.content_id = mi.content_id
|
||||
)
|
||||
OR (
|
||||
COALESCE(mi.default_metadata_language, '') <> ''
|
||||
AND LOWER(TRIM(mi.default_metadata_language))
|
||||
<> LOWER(TRIM(COALESCE(NULLIF(f.metadata_language, ''), 'en')))
|
||||
)
|
||||
)`
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
package adminjob
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// TestQuickRefreshListsLanguageMismatchedItems covers the listing half of
|
||||
// issue #211: quick-mode library refresh must include items whose stamped
|
||||
// default_metadata_language differs from the library's configured metadata
|
||||
// language, even when the item is otherwise complete (overview, poster,
|
||||
// backdrop all present) — otherwise a library language change never revisits
|
||||
// already-complete items.
|
||||
func TestQuickRefreshListsLanguageMismatchedItems(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect test database: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
mismatchID := fmt.Sprintf("lang-mismatch-%d", suffix)
|
||||
matchedID := fmt.Sprintf("lang-matched-%d", suffix)
|
||||
|
||||
var folderID int
|
||||
if err := pool.QueryRow(ctx, `
|
||||
INSERT INTO media_folders (type, name, enabled, metadata_language)
|
||||
VALUES ('movies', 'Lang Refresh Test', true, 'da')
|
||||
RETURNING id
|
||||
`).Scan(&folderID); err != nil {
|
||||
t.Fatalf("seed folder: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = ANY($1)`, []string{mismatchID, matchedID})
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_folders WHERE id = $1`, folderID)
|
||||
})
|
||||
|
||||
for _, row := range []struct {
|
||||
id, lang string
|
||||
}{
|
||||
{mismatchID, "zh"},
|
||||
{matchedID, "da"},
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_items (
|
||||
content_id, type, title, status, genres, tmdb_id,
|
||||
default_metadata_language, overview, poster_path, backdrop_path,
|
||||
last_refreshed, refresh_failures, episode_metadata_incomplete
|
||||
) VALUES ($1, 'movie', 'Complete Item', 'matched', '{}'::text[], '42',
|
||||
$2, 'An overview', '/p.jpg', '/b.jpg', NOW(), 0, FALSE)
|
||||
`, row.id, row.lang); err != nil {
|
||||
t.Fatalf("seed media item %s: %v", row.id, err)
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_item_libraries (content_id, media_folder_id)
|
||||
VALUES ($1, $2)
|
||||
`, row.id, folderID); err != nil {
|
||||
t.Fatalf("link media item %s: %v", row.id, err)
|
||||
}
|
||||
}
|
||||
|
||||
lister := NewPGLibraryRefreshItemLister(pool)
|
||||
items, err := lister.ListLibraryItems(ctx, folderID, LibraryRefreshModeQuick)
|
||||
if err != nil {
|
||||
t.Fatalf("ListLibraryItems: %v", err)
|
||||
}
|
||||
|
||||
var sawMismatch, sawMatched bool
|
||||
for _, item := range items {
|
||||
switch item.ContentID {
|
||||
case mismatchID:
|
||||
sawMismatch = true
|
||||
case matchedID:
|
||||
sawMatched = true
|
||||
}
|
||||
}
|
||||
if !sawMismatch {
|
||||
t.Errorf("quick refresh must include complete item with stamped language differing from the library language")
|
||||
}
|
||||
if sawMatched {
|
||||
t.Errorf("quick refresh must not include complete item whose stamped language matches the library language")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/api/handlers"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
)
|
||||
|
||||
// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB
|
||||
// conversion. Off by default; toggled via PUT /admin/settings/{key}.
|
||||
const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled"
|
||||
|
||||
// buildEbookConversion wires the in-process Kindle->EPUB converter for the read
|
||||
// handler. It compiles the embedded WASM module once at startup; if that fails
|
||||
// the feature stays off (returns nil) and the raw original is served as before.
|
||||
// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be
|
||||
// toggled without a restart yet does not hit the DB on every read.
|
||||
//
|
||||
// The Converter (a wazero runtime, ~no external resources — per-conversion
|
||||
// scratch dirs are cleaned as they go) is intentionally owned for the process
|
||||
// lifetime: it is not Close()d on shutdown because process exit reclaims it and
|
||||
// there is nothing to flush. Memory is bounded per conversion by
|
||||
// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's
|
||||
// resource notes); tune via ebookconvert.Options if the deployment is tight.
|
||||
func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion {
|
||||
if settings == nil {
|
||||
return nil
|
||||
}
|
||||
converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{})
|
||||
if err != nil {
|
||||
slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err)
|
||||
return nil
|
||||
}
|
||||
cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())})
|
||||
if err != nil {
|
||||
slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err)
|
||||
_ = converter.Close(context.Background())
|
||||
return nil
|
||||
}
|
||||
slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey)
|
||||
return &handlers.EbookConversion{
|
||||
Converter: cache,
|
||||
Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey),
|
||||
}
|
||||
}
|
||||
|
||||
// ebookFlagPredicate returns a cached boolean reader for a server setting. The
|
||||
// read path (every ebook read + the capability endpoint, which clients may
|
||||
// poll) would otherwise issue a DB query per request; a short TTL collapses
|
||||
// bursts to one query while keeping admin toggles effectively instant. On a read
|
||||
// error it keeps the last known value rather than flapping the feature off.
|
||||
func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool {
|
||||
const ttl = 3 * time.Second
|
||||
var (
|
||||
mu sync.Mutex
|
||||
value bool
|
||||
fetched time.Time
|
||||
hasVal bool
|
||||
)
|
||||
return func(ctx context.Context) bool {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if hasVal && time.Since(fetched) < ttl {
|
||||
return value
|
||||
}
|
||||
v, err := settings.Get(ctx, key)
|
||||
if err != nil {
|
||||
return value // keep last known (false until the first successful read)
|
||||
}
|
||||
value = isTruthySetting(v)
|
||||
fetched = time.Now()
|
||||
hasVal = true
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling
|
||||
// of the transcode dir so it lives alongside other derived media.
|
||||
func ebookConversionCacheDir(cfg *config.Config) string {
|
||||
base := os.TempDir()
|
||||
if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" {
|
||||
base = filepath.Dir(cfg.Playback.TranscodeDir)
|
||||
}
|
||||
return filepath.Join(base, "silo-ebook-epub")
|
||||
}
|
||||
|
||||
func isTruthySetting(v string) bool {
|
||||
switch strings.ToLower(strings.TrimSpace(v)) {
|
||||
case "true", "1", "yes", "on", "enabled":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// stubSettings counts Get calls so we can assert the flag predicate caches.
|
||||
type stubSettings struct {
|
||||
value string
|
||||
gets int64
|
||||
}
|
||||
|
||||
func (s *stubSettings) Get(_ context.Context, _ string) (string, error) {
|
||||
atomic.AddInt64(&s.gets, 1)
|
||||
return s.value, nil
|
||||
}
|
||||
func (s *stubSettings) Set(context.Context, string, string) error { return nil }
|
||||
func (s *stubSettings) GetAll(context.Context) (map[string]string, error) {
|
||||
return map[string]string{}, nil
|
||||
}
|
||||
|
||||
// The flag predicate must not hit the settings store on every call: the read
|
||||
// path (every ebook read + capability poll) would otherwise issue a DB query
|
||||
// per request.
|
||||
func TestEbookFlagPredicate_CachesWithinTTL(t *testing.T) {
|
||||
s := &stubSettings{value: "true"}
|
||||
pred := ebookFlagPredicate(s, "ebook.kindle_conversion_enabled")
|
||||
|
||||
for i := 0; i < 50; i++ {
|
||||
if !pred(context.Background()) {
|
||||
t.Fatalf("call %d: predicate returned false for a truthy flag", i)
|
||||
}
|
||||
}
|
||||
if got := atomic.LoadInt64(&s.gets); got > 2 {
|
||||
t.Fatalf("settings.Get called %d times across 50 predicate calls; expected it to be cached", got)
|
||||
}
|
||||
}
|
||||
@@ -8,7 +8,9 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"math"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"slices"
|
||||
"sort"
|
||||
"strconv"
|
||||
@@ -99,6 +101,7 @@ type AdminHandler struct {
|
||||
OnUserSessionsRevoked func(ctx context.Context, userID int)
|
||||
OnServerSettingUpdated func(ctx context.Context, key, value string)
|
||||
RestartStatus *ServerRestartStatusTracker
|
||||
CatalogSearchStatus catalog.CatalogSearchStatusProvider
|
||||
}
|
||||
|
||||
// NewAdminHandler creates a new AdminHandler backed by the given
|
||||
@@ -2138,6 +2141,109 @@ func (h *AdminHandler) HandleUpdateSetting(w http.ResponseWriter, r *http.Reques
|
||||
"subtitle_ai.transcribe_quota_period must be day, week, or month")
|
||||
return
|
||||
}
|
||||
case notifications.SettingApplePushDeliveryEnabled:
|
||||
enabled, err := strconv.ParseBool(strings.TrimSpace(req.Value))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "notifications.apple_push_delivery_enabled must be true or false")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.FormatBool(enabled)
|
||||
case notifications.SettingPushRelayURL, notifications.SettingPushRelayDeploymentID, notifications.SettingPushRelayAPIKey:
|
||||
// The registration flow persists the relay URL, deployment id, and API
|
||||
// key together; a direct write to any of them desyncs the stored URL
|
||||
// from the credentials the relay minted for it (and feeds an arbitrary
|
||||
// id into the next rotation request).
|
||||
writeError(w, http.StatusBadRequest, "bad_request",
|
||||
key+" is managed by the push relay registration flow; use POST /admin/notifications/push/relay/register")
|
||||
return
|
||||
case catalog.SearchSettingProvider:
|
||||
switch strings.TrimSpace(strings.ToLower(req.Value)) {
|
||||
case catalog.SearchProviderPostgres, catalog.SearchProviderMeilisearch:
|
||||
req.Value = strings.TrimSpace(strings.ToLower(req.Value))
|
||||
default:
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.provider must be postgres or meilisearch")
|
||||
return
|
||||
}
|
||||
case catalog.SearchSettingMeilisearchURL:
|
||||
value := strings.TrimSpace(req.Value)
|
||||
if value != "" {
|
||||
parsed, err := url.Parse(value)
|
||||
if err != nil || parsed.Scheme == "" || parsed.Host == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.url must include scheme and host")
|
||||
return
|
||||
}
|
||||
}
|
||||
req.Value = value
|
||||
case catalog.SearchSettingMeilisearchIndex:
|
||||
req.Value = strings.TrimSpace(req.Value)
|
||||
if req.Value == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.index is required")
|
||||
return
|
||||
}
|
||||
case catalog.SearchSettingMeilisearchTimeoutMS:
|
||||
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
|
||||
if err != nil || n <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.timeout_ms must be an integer greater than 0")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.Itoa(n)
|
||||
case catalog.SearchSettingMeilisearchMatchingStrategy:
|
||||
switch strings.TrimSpace(strings.ToLower(req.Value)) {
|
||||
case "last", "all":
|
||||
req.Value = strings.TrimSpace(strings.ToLower(req.Value))
|
||||
default:
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.matching_strategy must be last or all")
|
||||
return
|
||||
}
|
||||
case catalog.SearchSettingMeilisearchSyncBatchSize:
|
||||
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
|
||||
if err != nil || n < 1 || n > catalog.MaxMeilisearchSyncBatchSize {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.sync_batch_size must be an integer between 1 and 10000")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.Itoa(n)
|
||||
case catalog.SearchSettingMeilisearchRebuildBatchSize:
|
||||
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
|
||||
if err != nil || n < 1 || n > catalog.MaxMeilisearchRebuildBatchSize {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.rebuild_batch_size must be an integer between 1 and 25000")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.Itoa(n)
|
||||
case catalog.SearchSettingMeilisearchRebuildQueue:
|
||||
n, err := strconv.Atoi(strings.TrimSpace(req.Value))
|
||||
if err != nil || n < 1 || n > catalog.MaxMeilisearchRebuildQueueDepth {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.rebuild_task_queue_depth must be an integer between 1 and 16")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.Itoa(n)
|
||||
case catalog.SearchSettingMeilisearchIndexTypes:
|
||||
itemTypes, err := catalog.NormalizeCatalogSearchIndexTypesValue(req.Value)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
req.Value = catalog.FormatCatalogSearchIndexTypesValue(itemTypes)
|
||||
case catalog.SearchSettingMeilisearchSemanticEnabled:
|
||||
enabled, err := strconv.ParseBool(strings.TrimSpace(req.Value))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.semantic_enabled must be true or false")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.FormatBool(enabled)
|
||||
case catalog.SearchSettingMeilisearchSemanticRatio:
|
||||
ratio, err := strconv.ParseFloat(strings.TrimSpace(req.Value), 64)
|
||||
if err != nil || math.IsNaN(ratio) || ratio < 0 || ratio > 1 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "catalog.search.meilisearch.semantic_ratio must be a number between 0 and 1")
|
||||
return
|
||||
}
|
||||
req.Value = strconv.FormatFloat(ratio, 'f', -1, 64)
|
||||
case catalog.SearchSettingMeilisearchEmbedder:
|
||||
embedder, err := catalog.NormalizeCatalogSearchEmbedderName(req.Value)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
req.Value = embedder
|
||||
}
|
||||
|
||||
if err := h.SettingsRepo.Set(r.Context(), key, req.Value); err != nil {
|
||||
|
||||
@@ -0,0 +1,279 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
type AdminApplePushHandler struct {
|
||||
system *notifications.System
|
||||
settings ServerSettingsStore
|
||||
client httpDoer
|
||||
}
|
||||
|
||||
type httpDoer interface {
|
||||
Do(*http.Request) (*http.Response, error)
|
||||
}
|
||||
|
||||
func NewAdminApplePushHandler(system *notifications.System, settings ServerSettingsStore) *AdminApplePushHandler {
|
||||
return &AdminApplePushHandler{
|
||||
system: system,
|
||||
settings: settings,
|
||||
client: &http.Client{Timeout: 10 * time.Second},
|
||||
}
|
||||
}
|
||||
|
||||
type adminApplePushTestRequest struct {
|
||||
ProfileID string `json:"profile_id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
}
|
||||
|
||||
type adminApplePushTestResponse struct {
|
||||
AttemptID string `json:"attempt_id"`
|
||||
PushDeviceID string `json:"push_device_id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
Outcome string `json:"outcome"`
|
||||
RelayRequestID string `json:"relay_request_id,omitempty"`
|
||||
UpstreamStatus *int `json:"upstream_status,omitempty"`
|
||||
UpstreamReason string `json:"upstream_reason,omitempty"`
|
||||
FailureMessage string `json:"failure_message,omitempty"`
|
||||
}
|
||||
|
||||
type adminPushRelayRegisterRequest struct {
|
||||
RelayURL string `json:"relay_url"`
|
||||
}
|
||||
|
||||
type pushRelayRegisterRequest struct {
|
||||
DeploymentID string `json:"deployment_id,omitempty"`
|
||||
}
|
||||
|
||||
type pushRelayRegisterResponse struct {
|
||||
RequestID string `json:"request_id"`
|
||||
DeploymentID string `json:"deployment_id"`
|
||||
APIKey string `json:"api_key"`
|
||||
KeyPrefix string `json:"key_prefix"`
|
||||
APNsTopics []string `json:"apns_topics"`
|
||||
}
|
||||
|
||||
type pushRelayNestedError struct {
|
||||
Error struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
RequestID string `json:"request_id"`
|
||||
} `json:"error"`
|
||||
}
|
||||
|
||||
type adminPushRelayRegisterResponse struct {
|
||||
RelayURL string `json:"relay_url"`
|
||||
DeploymentID string `json:"deployment_id"`
|
||||
KeyPrefix string `json:"key_prefix"`
|
||||
APIKeyConfigured bool `json:"api_key_configured"`
|
||||
RelayRequestID string `json:"relay_request_id,omitempty"`
|
||||
APNsTopics []string `json:"apns_topics,omitempty"`
|
||||
}
|
||||
|
||||
// HandleTest handles POST /admin/notifications/push/apple/test.
|
||||
func (h *AdminApplePushHandler) HandleTest(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.system == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push delivery is not available")
|
||||
return
|
||||
}
|
||||
var req adminApplePushTestRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
result, err := h.system.SendApplePushTest(r.Context(), req.ProfileID, req.ServerDeviceID)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, notifications.ErrPushDeliveryInvalid):
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeliveryNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Apple push device not found")
|
||||
case errors.Is(err, notifications.ErrPushDeliveryUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push delivery is not available")
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to send Apple push test")
|
||||
}
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, adminApplePushTestResponse{
|
||||
AttemptID: result.AttemptID,
|
||||
PushDeviceID: result.PushDeviceID,
|
||||
ServerDeviceID: result.ServerDeviceID,
|
||||
Outcome: result.Outcome,
|
||||
RelayRequestID: result.RelayRequestID,
|
||||
UpstreamStatus: result.UpstreamStatus,
|
||||
UpstreamReason: result.UpstreamReason,
|
||||
FailureMessage: result.FailureMessage,
|
||||
})
|
||||
}
|
||||
|
||||
// HandleRegisterRelay handles POST /admin/notifications/push/relay/register.
|
||||
func (h *AdminApplePushHandler) HandleRegisterRelay(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.settings == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Settings store is not available")
|
||||
return
|
||||
}
|
||||
var req adminPushRelayRegisterRequest
|
||||
dec := json.NewDecoder(r.Body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
relayURL, err := normalizePushRelayURL(req.RelayURL)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
existingDeploymentID, err := h.settings.Get(r.Context(), notifications.SettingPushRelayDeploymentID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to load relay deployment id")
|
||||
return
|
||||
}
|
||||
relayResp, err := h.registerWithRelay(r, relayURL, strings.TrimSpace(existingDeploymentID))
|
||||
if err != nil {
|
||||
status, code, message := mapRelayRegistrationError(err)
|
||||
writeError(w, status, code, message)
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(relayResp.DeploymentID) == "" || strings.TrimSpace(relayResp.APIKey) == "" {
|
||||
writeError(w, http.StatusBadGateway, "relay_bad_response", "Push relay returned an incomplete registration response")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayURL, relayURL); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay URL")
|
||||
return
|
||||
}
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayDeploymentID, relayResp.DeploymentID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay deployment id")
|
||||
return
|
||||
}
|
||||
if err := h.settings.Set(r.Context(), notifications.SettingPushRelayAPIKey, relayResp.APIKey); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "settings_error", "Failed to save push relay API key")
|
||||
return
|
||||
}
|
||||
if h.system != nil && h.system.Settings != nil {
|
||||
h.system.Settings.Invalidate(
|
||||
notifications.SettingPushRelayURL,
|
||||
notifications.SettingPushRelayDeploymentID,
|
||||
notifications.SettingPushRelayAPIKey,
|
||||
)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, adminPushRelayRegisterResponse{
|
||||
RelayURL: relayURL,
|
||||
DeploymentID: relayResp.DeploymentID,
|
||||
KeyPrefix: relayResp.KeyPrefix,
|
||||
APIKeyConfigured: true,
|
||||
RelayRequestID: relayResp.RequestID,
|
||||
APNsTopics: relayResp.APNsTopics,
|
||||
})
|
||||
}
|
||||
|
||||
func (h *AdminApplePushHandler) registerWithRelay(r *http.Request, relayURL, deploymentID string) (*pushRelayRegisterResponse, error) {
|
||||
body, err := json.Marshal(pushRelayRegisterRequest{
|
||||
DeploymentID: deploymentID,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
httpReq, err := http.NewRequestWithContext(r.Context(), http.MethodPost, relayURL+"/v1/deployments/register", bytes.NewReader(body))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
httpReq.Header.Set("Content-Type", "application/json")
|
||||
httpReq.Header.Set("User-Agent", "Silo-Server/PushRelayRegistration")
|
||||
|
||||
resp, err := h.client.Do(httpReq)
|
||||
if err != nil {
|
||||
return nil, relayRegistrationError{status: http.StatusBadGateway, code: "relay_unreachable", message: "Push relay could not be reached"}
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
|
||||
data, _ := io.ReadAll(io.LimitReader(resp.Body, 16<<10))
|
||||
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
|
||||
var parsed pushRelayNestedError
|
||||
_ = json.Unmarshal(data, &parsed)
|
||||
code := parsed.Error.Code
|
||||
if code == "" {
|
||||
code = fmt.Sprintf("relay_http_%d", resp.StatusCode)
|
||||
}
|
||||
message := parsed.Error.Message
|
||||
if message == "" {
|
||||
message = http.StatusText(resp.StatusCode)
|
||||
}
|
||||
return nil, relayRegistrationError{status: resp.StatusCode, code: code, message: message}
|
||||
}
|
||||
var parsed pushRelayRegisterResponse
|
||||
if err := json.Unmarshal(data, &parsed); err != nil {
|
||||
return nil, relayRegistrationError{status: http.StatusBadGateway, code: "relay_bad_response", message: "Push relay returned invalid JSON"}
|
||||
}
|
||||
return &parsed, nil
|
||||
}
|
||||
|
||||
type relayRegistrationError struct {
|
||||
status int
|
||||
code string
|
||||
message string
|
||||
}
|
||||
|
||||
func (e relayRegistrationError) Error() string {
|
||||
return e.code
|
||||
}
|
||||
|
||||
func mapRelayRegistrationError(err error) (int, string, string) {
|
||||
var relayErr relayRegistrationError
|
||||
if !errors.As(err, &relayErr) {
|
||||
return http.StatusInternalServerError, "internal_error", "Failed to register push relay"
|
||||
}
|
||||
switch relayErr.status {
|
||||
case http.StatusForbidden:
|
||||
return http.StatusUnprocessableEntity, "relay_deployment_rejected", "Push relay rejected this deployment"
|
||||
case http.StatusTooManyRequests:
|
||||
return http.StatusTooManyRequests, "relay_rate_limited", relayErr.message
|
||||
case http.StatusServiceUnavailable:
|
||||
return http.StatusServiceUnavailable, "relay_unavailable", relayErr.message
|
||||
default:
|
||||
if relayErr.status >= 500 {
|
||||
return http.StatusBadGateway, "relay_error", relayErr.message
|
||||
}
|
||||
return http.StatusBadGateway, relayErr.code, relayErr.message
|
||||
}
|
||||
}
|
||||
|
||||
// normalizePushRelayURLValue trims a relay URL and enforces https with a
|
||||
// host. Empty input stays empty so callers choose their own default.
|
||||
func normalizePushRelayURLValue(raw string) (string, error) {
|
||||
value := strings.TrimRight(strings.TrimSpace(raw), "/")
|
||||
if value == "" {
|
||||
return "", nil
|
||||
}
|
||||
parsed, err := url.Parse(value)
|
||||
if err != nil || parsed.Scheme != "https" || parsed.Host == "" {
|
||||
return "", errors.New("relay_url must be an https URL")
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
func normalizePushRelayURL(raw string) (string, error) {
|
||||
value, err := normalizePushRelayURLValue(raw)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if value == "" {
|
||||
return notifications.DefaultPushRelayURL, nil
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
func TestAdminApplePushHandlerUnavailableWithoutSystem(t *testing.T) {
|
||||
h := NewAdminApplePushHandler(nil, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleTest(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/apple/test", strings.NewReader(`{}`)))
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("HandleTest without system = %d, want 503", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerRejectsInvalidJSON(t *testing.T) {
|
||||
h := NewAdminApplePushHandler(¬ifications.System{}, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleTest(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/apple/test", strings.NewReader(`{`)))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("HandleTest invalid JSON = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerRegistersRelayAndStoresKey(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{values: map[string]string{
|
||||
notifications.SettingPushRelayDeploymentID: "01EXISTING",
|
||||
}}
|
||||
var relayReq map[string]string
|
||||
relay := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/v1/deployments/register" {
|
||||
t.Fatalf("relay path = %s", r.URL.Path)
|
||||
}
|
||||
if err := json.NewDecoder(r.Body).Decode(&relayReq); err != nil {
|
||||
t.Fatalf("decode relay request: %v", err)
|
||||
}
|
||||
writeJSON(w, http.StatusOK, pushRelayRegisterResponse{
|
||||
RequestID: "relay-request",
|
||||
DeploymentID: "01RETURNED",
|
||||
APIKey: "rk_live_raw-key",
|
||||
KeyPrefix: "rk_live_raw",
|
||||
APNsTopics: []string{"org.siloserver.silo"},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(relay.Close)
|
||||
|
||||
h := NewAdminApplePushHandler(¬ifications.System{Settings: notifications.NewSettings(settings)}, settings)
|
||||
h.client = relay.Client()
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+relay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("http relay URL status = %d, want 400 because HTTPS is required", rec.Code)
|
||||
}
|
||||
|
||||
httpsRelay := httptest.NewTLSServer(relay.Config.Handler)
|
||||
t.Cleanup(httpsRelay.Close)
|
||||
h.client = httpsRelay.Client()
|
||||
rec = httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+httpsRelay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d (%s), want 200", rec.Code, rec.Body.String())
|
||||
}
|
||||
if relayReq["deployment_id"] != "01EXISTING" || len(relayReq) != 1 {
|
||||
t.Fatalf("relay request = %+v", relayReq)
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayURL] != httpsRelay.URL {
|
||||
t.Fatalf("stored relay URL = %q", settings.values[notifications.SettingPushRelayURL])
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayDeploymentID] != "01RETURNED" {
|
||||
t.Fatalf("stored deployment id = %q", settings.values[notifications.SettingPushRelayDeploymentID])
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayAPIKey] != "rk_live_raw-key" {
|
||||
t.Fatalf("stored api key = %q", settings.values[notifications.SettingPushRelayAPIKey])
|
||||
}
|
||||
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if _, ok := body["api_key"]; ok {
|
||||
t.Fatal("response leaked raw api_key")
|
||||
}
|
||||
if body["api_key_configured"] != true || body["deployment_id"] != "01RETURNED" {
|
||||
t.Fatalf("response = %+v", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminApplePushHandlerMapsRelayRateLimit(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{values: map[string]string{}}
|
||||
relay := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusTooManyRequests, map[string]any{
|
||||
"error": map[string]string{"code": "rate_limited", "message": "too many deployment registrations from this network"},
|
||||
})
|
||||
}))
|
||||
t.Cleanup(relay.Close)
|
||||
|
||||
h := NewAdminApplePushHandler(¬ifications.System{}, settings)
|
||||
h.client = relay.Client()
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleRegisterRelay(rec, httptest.NewRequest(http.MethodPost, "/admin/notifications/push/relay/register", strings.NewReader(`{
|
||||
"relay_url":"`+relay.URL+`"
|
||||
}`)))
|
||||
|
||||
if rec.Code != http.StatusTooManyRequests {
|
||||
t.Fatalf("status = %d (%s), want 429", rec.Code, rec.Body.String())
|
||||
}
|
||||
if settings.values[notifications.SettingPushRelayAPIKey] != "" {
|
||||
t.Fatal("api key was stored after relay rejected registration")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
)
|
||||
|
||||
func (h *AdminHandler) HandleGetCatalogSearchStatus(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.CatalogSearchStatus == nil {
|
||||
writeJSON(w, http.StatusOK, catalog.CatalogSearchRuntimeStatus{
|
||||
ConfiguredProvider: catalog.SearchProviderPostgres,
|
||||
ActiveProvider: catalog.SearchProviderPostgres,
|
||||
Meilisearch: catalog.CatalogSearchMeiliStatus{
|
||||
Configured: false,
|
||||
Healthy: false,
|
||||
CircuitState: "not_configured",
|
||||
},
|
||||
Index: catalog.CatalogSearchIndexStateStatus{
|
||||
ExpectedSchemaVersion: catalog.SearchMeilisearchSchemaVersion,
|
||||
},
|
||||
Tasks: []catalog.CatalogSearchTaskLink{
|
||||
{Key: "sync_catalog_search_index", Name: "Sync Catalog Search Index", Href: "/admin/tasks/sync_catalog_search_index"},
|
||||
{Key: "rebuild_catalog_search_index", Name: "Rebuild Catalog Search Index", Href: "/admin/tasks/rebuild_catalog_search_index"},
|
||||
},
|
||||
})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, h.CatalogSearchStatus.Status(r.Context()))
|
||||
}
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
@@ -56,12 +57,13 @@ func NewAdminMatchHandler(
|
||||
// --- Request/Response types ---
|
||||
|
||||
type matchSearchRequest struct {
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year"`
|
||||
ImdbID string `json:"imdb_id"`
|
||||
TmdbID string `json:"tmdb_id"`
|
||||
TvdbID string `json:"tvdb_id"`
|
||||
LibraryID *int `json:"library_id,omitempty"`
|
||||
Title string `json:"title"`
|
||||
Year int `json:"year"`
|
||||
ImdbID string `json:"imdb_id"`
|
||||
TmdbID string `json:"tmdb_id"`
|
||||
TvdbID string `json:"tvdb_id"`
|
||||
ProviderIDs map[string]string `json:"provider_ids,omitempty"`
|
||||
LibraryID *int `json:"library_id,omitempty"`
|
||||
}
|
||||
|
||||
type matchSearchResponse struct {
|
||||
@@ -119,7 +121,7 @@ func (h *AdminMatchHandler) HandleSearchItemMatchCandidates(w http.ResponseWrite
|
||||
Title: req.Title,
|
||||
Year: req.Year,
|
||||
ContentType: item.Type,
|
||||
ProviderIDs: make(map[string]string),
|
||||
ProviderIDs: normalizeMatchProviderIDs(req.ProviderIDs),
|
||||
}
|
||||
if query.Title == "" {
|
||||
query.Title = item.Title
|
||||
@@ -129,15 +131,9 @@ func (h *AdminMatchHandler) HandleSearchItemMatchCandidates(w http.ResponseWrite
|
||||
}
|
||||
|
||||
// Inject any provider IDs from the request.
|
||||
if req.ImdbID != "" {
|
||||
query.ProviderIDs["imdb"] = req.ImdbID
|
||||
}
|
||||
if req.TmdbID != "" {
|
||||
query.ProviderIDs["tmdb"] = req.TmdbID
|
||||
}
|
||||
if req.TvdbID != "" {
|
||||
query.ProviderIDs["tvdb"] = req.TvdbID
|
||||
}
|
||||
setMatchProviderID(query.ProviderIDs, "imdb", req.ImdbID)
|
||||
setMatchProviderID(query.ProviderIDs, "tmdb", req.TmdbID)
|
||||
setMatchProviderID(query.ProviderIDs, "tvdb", req.TvdbID)
|
||||
|
||||
candidates, err := h.metadata.SearchAndNormalize(r.Context(), query, folderID)
|
||||
if err != nil {
|
||||
@@ -165,7 +161,11 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
|
||||
return
|
||||
}
|
||||
|
||||
if len(req.ProviderIDs) == 0 {
|
||||
// Normalize keys and values the same way the search endpoint does so a
|
||||
// caller-supplied key like "TMDB" or " tmdb " cannot bypass downstream
|
||||
// provider-id handling (e.g. stale-ID suppression in metadata.Process).
|
||||
providerIDs := normalizeMatchProviderIDs(req.ProviderIDs)
|
||||
if len(providerIDs) == 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "At least one provider ID is required")
|
||||
return
|
||||
}
|
||||
@@ -195,7 +195,7 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
|
||||
|
||||
result, err := h.metadata.Process(r.Context(), metadata.ProcessRequest{
|
||||
ContentID: contentID,
|
||||
ProviderIDs: req.ProviderIDs,
|
||||
ProviderIDs: providerIDs,
|
||||
FolderID: folderIDStr,
|
||||
Mode: metadata.ModeIdentify,
|
||||
})
|
||||
@@ -211,6 +211,23 @@ func (h *AdminMatchHandler) HandleApplyItemMatch(w http.ResponseWriter, r *http.
|
||||
})
|
||||
}
|
||||
|
||||
func normalizeMatchProviderIDs(input map[string]string) map[string]string {
|
||||
out := make(map[string]string, len(input))
|
||||
for provider, providerID := range input {
|
||||
setMatchProviderID(out, provider, providerID)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func setMatchProviderID(providerIDs map[string]string, provider string, providerID string) {
|
||||
provider = strings.ToLower(strings.TrimSpace(provider))
|
||||
providerID = strings.TrimSpace(providerID)
|
||||
if provider == "" || providerID == "" {
|
||||
return
|
||||
}
|
||||
providerIDs[provider] = providerID
|
||||
}
|
||||
|
||||
// PoolFolderLookup implements MatchFolderLookup using a direct pgx pool query
|
||||
// against the media_item_libraries table.
|
||||
type PoolFolderLookup struct {
|
||||
|
||||
@@ -199,6 +199,58 @@ func TestAdminMatchSearch_FallsBackToItemMetadata(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchSearch_AcceptsGenericProviderIDsAndLibraryID(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
"ebook-1": {ContentID: "ebook-1", Title: "Example Book", Year: 2024, Type: "ebook"},
|
||||
},
|
||||
}
|
||||
folders := &fakeMatchFolderLookup{
|
||||
folderIDs: map[string][]int{"ebook-1": {7, 8}},
|
||||
}
|
||||
captureSvc := &capturingMetadataService{
|
||||
inner: &fakeMatchMetadataService{searchResults: []metadata.MatchCandidate{}},
|
||||
}
|
||||
|
||||
h := NewAdminMatchHandler(items, folders, captureSvc)
|
||||
router := buildMatchRouter(h)
|
||||
|
||||
libraryID := 8
|
||||
body, _ := json.Marshal(matchSearchRequest{
|
||||
ProviderIDs: map[string]string{
|
||||
" ISBN ": " 9780000000001 ",
|
||||
"goodreads": " ",
|
||||
},
|
||||
TmdbID: "12345",
|
||||
LibraryID: &libraryID,
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/admin/items/ebook-1/match/search", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
if captureSvc.lastFolderID != 8 {
|
||||
t.Fatalf("folderID = %d, want 8", captureSvc.lastFolderID)
|
||||
}
|
||||
if captureSvc.lastQuery.ContentType != "ebook" {
|
||||
t.Fatalf("ContentType = %q, want ebook", captureSvc.lastQuery.ContentType)
|
||||
}
|
||||
if got := captureSvc.lastQuery.ProviderIDs["isbn"]; got != "9780000000001" {
|
||||
t.Fatalf("ProviderIDs[isbn] = %q, want 9780000000001", got)
|
||||
}
|
||||
if got := captureSvc.lastQuery.ProviderIDs["tmdb"]; got != "12345" {
|
||||
t.Fatalf("ProviderIDs[tmdb] = %q, want 12345", got)
|
||||
}
|
||||
if _, ok := captureSvc.lastQuery.ProviderIDs["goodreads"]; ok {
|
||||
t.Fatalf("blank provider IDs should be ignored, got %v", captureSvc.lastQuery.ProviderIDs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_PreservesContentID(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
@@ -274,6 +326,67 @@ func TestAdminMatchApply_RequiresProviderIDs(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_NormalizesProviderIDKeys(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
"item-5": {ContentID: "item-5", Title: "Test", Type: "movie"},
|
||||
},
|
||||
}
|
||||
metaSvc := &fakeMatchMetadataService{
|
||||
processResult: &metadata.ProcessResult{ContentID: "item-5", Updated: true},
|
||||
}
|
||||
h := NewAdminMatchHandler(items, nil, metaSvc)
|
||||
router := buildMatchRouter(h)
|
||||
|
||||
body, _ := json.Marshal(matchApplyRequest{
|
||||
ProviderIDs: map[string]string{
|
||||
" TMDB ": " 157336 ",
|
||||
"blank": " ",
|
||||
},
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/admin/items/item-5/match/apply", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("expected 200, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := metaSvc.lastProcess.ProviderIDs["tmdb"]; got != "157336" {
|
||||
t.Errorf("Process provider_ids[tmdb] = %q, want 157336 (keys/values must be normalized)", got)
|
||||
}
|
||||
if _, ok := metaSvc.lastProcess.ProviderIDs[" TMDB "]; ok {
|
||||
t.Errorf("Process received raw key %q, want normalized keys only: %v", " TMDB ", metaSvc.lastProcess.ProviderIDs)
|
||||
}
|
||||
if _, ok := metaSvc.lastProcess.ProviderIDs["blank"]; ok {
|
||||
t.Errorf("blank provider IDs should be dropped, got %v", metaSvc.lastProcess.ProviderIDs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_RejectsAllBlankProviderIDs(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{
|
||||
items: map[string]*models.MediaItem{
|
||||
"item-6": {ContentID: "item-6", Title: "Test", Type: "movie"},
|
||||
},
|
||||
}
|
||||
h := NewAdminMatchHandler(items, nil, &fakeMatchMetadataService{})
|
||||
router := buildMatchRouter(h)
|
||||
|
||||
body, _ := json.Marshal(matchApplyRequest{
|
||||
ProviderIDs: map[string]string{"tmdb": " ", " ": "123"},
|
||||
})
|
||||
req := httptest.NewRequest(http.MethodPost, "/admin/items/item-6/match/apply", bytes.NewReader(body))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
router.ServeHTTP(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("expected 400 for all-blank provider_ids, got %d: %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminMatchApply_ItemNotFound(t *testing.T) {
|
||||
items := &fakeMatchItemLookup{items: map[string]*models.MediaItem{}}
|
||||
h := NewAdminMatchHandler(items, nil, &fakeMatchMetadataService{})
|
||||
@@ -293,12 +406,14 @@ func TestAdminMatchApply_ItemNotFound(t *testing.T) {
|
||||
|
||||
// capturingMetadataService wraps a fake to capture the query passed to SearchAndNormalize.
|
||||
type capturingMetadataService struct {
|
||||
inner *fakeMatchMetadataService
|
||||
lastQuery metadata.SearchQuery
|
||||
inner *fakeMatchMetadataService
|
||||
lastQuery metadata.SearchQuery
|
||||
lastFolderID int
|
||||
}
|
||||
|
||||
func (c *capturingMetadataService) SearchAndNormalize(ctx context.Context, query metadata.SearchQuery, folderID int) ([]metadata.MatchCandidate, error) {
|
||||
c.lastQuery = query
|
||||
c.lastFolderID = folderID
|
||||
return c.inner.SearchAndNormalize(ctx, query, folderID)
|
||||
}
|
||||
|
||||
|
||||
@@ -179,7 +179,7 @@ func (h *AdminPlaybackControlHandler) handleSessionCommand(w http.ResponseWriter
|
||||
|
||||
fallback := func() {
|
||||
h.playback.forgetRealtimeCommand(commandID)
|
||||
_ = h.playback.stopPlaybackSessionByID(context.Background(), sessionID)
|
||||
_ = h.playback.stopPlaybackSessionByID(context.Background(), sessionID, true)
|
||||
}
|
||||
|
||||
h.playback.rememberRealtimeCommand(commandID, sessionID, name)
|
||||
|
||||
@@ -41,6 +41,8 @@ type serverChannelResponse struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
NotifyNewMovies bool `json:"notify_new_movies"`
|
||||
NotifyNewEpisodes bool `json:"notify_new_episodes"`
|
||||
NotifyNewAudiobooks bool `json:"notify_new_audiobooks"`
|
||||
NotifyNewEbooks bool `json:"notify_new_ebooks"`
|
||||
NotifyRequestSubmitted bool `json:"notify_request_submitted"`
|
||||
NotifyRequestApproved bool `json:"notify_request_approved"`
|
||||
NotifyRequestDeclined bool `json:"notify_request_declined"`
|
||||
@@ -65,6 +67,8 @@ func serverChannelToResponse(ch notifications.ServerChannel) serverChannelRespon
|
||||
Enabled: ch.Enabled,
|
||||
NotifyNewMovies: ch.NotifyNewMovies,
|
||||
NotifyNewEpisodes: ch.NotifyNewEpisodes,
|
||||
NotifyNewAudiobooks: ch.NotifyNewAudiobooks,
|
||||
NotifyNewEbooks: ch.NotifyNewEbooks,
|
||||
NotifyRequestSubmitted: ch.NotifyRequestSubmitted,
|
||||
NotifyRequestApproved: ch.NotifyRequestApproved,
|
||||
NotifyRequestDeclined: ch.NotifyRequestDeclined,
|
||||
@@ -86,6 +90,8 @@ type serverChannelRequest struct {
|
||||
Enabled *bool `json:"enabled"`
|
||||
NotifyNewMovies *bool `json:"notify_new_movies"`
|
||||
NotifyNewEpisodes *bool `json:"notify_new_episodes"`
|
||||
NotifyNewAudiobooks *bool `json:"notify_new_audiobooks"`
|
||||
NotifyNewEbooks *bool `json:"notify_new_ebooks"`
|
||||
NotifyRequestSubmitted *bool `json:"notify_request_submitted"`
|
||||
NotifyRequestApproved *bool `json:"notify_request_approved"`
|
||||
NotifyRequestDeclined *bool `json:"notify_request_declined"`
|
||||
@@ -100,6 +106,8 @@ func (r serverChannelRequest) toInput() notifications.ServerChannelInput {
|
||||
Enabled: r.Enabled,
|
||||
NotifyNewMovies: r.NotifyNewMovies,
|
||||
NotifyNewEpisodes: r.NotifyNewEpisodes,
|
||||
NotifyNewAudiobooks: r.NotifyNewAudiobooks,
|
||||
NotifyNewEbooks: r.NotifyNewEbooks,
|
||||
NotifyRequestSubmitted: r.NotifyRequestSubmitted,
|
||||
NotifyRequestApproved: r.NotifyRequestApproved,
|
||||
NotifyRequestDeclined: r.NotifyRequestDeclined,
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"github.com/redis/go-redis/v9"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/cache"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/recommendations/embeddings"
|
||||
"github.com/Silo-Server/silo-server/internal/s3client"
|
||||
@@ -116,6 +117,8 @@ func (h *AdminHandler) HandleCheckSettingsConnection(w http.ResponseWriter, r *h
|
||||
response = checkRedisConnection(r.Context(), cfg)
|
||||
case "recommendations_embedding":
|
||||
response = checkRecommendationsEmbeddingConnection(r.Context(), cfg)
|
||||
case "meilisearch":
|
||||
response = checkMeilisearchConnection(r.Context(), effectiveSettings)
|
||||
default:
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Unsupported connection check kind")
|
||||
return
|
||||
@@ -124,6 +127,22 @@ func (h *AdminHandler) HandleCheckSettingsConnection(w http.ResponseWriter, r *h
|
||||
writeJSON(w, http.StatusOK, response)
|
||||
}
|
||||
|
||||
func checkMeilisearchConnection(ctx context.Context, settings map[string]string) connectionCheckResponse {
|
||||
searchSettings, err := catalog.CatalogSearchSettingsFromMap(settings)
|
||||
if err != nil {
|
||||
return connectionCheckResponse{Success: false, Message: err.Error()}
|
||||
}
|
||||
if searchSettings.MeilisearchURL == "" {
|
||||
return connectionCheckResponse{Success: false, Message: "Meilisearch URL is required"}
|
||||
}
|
||||
searchSettings.Provider = catalog.SearchProviderMeilisearch
|
||||
indexer := catalog.NewCatalogSearchIndexer(nil, nil)
|
||||
if err := indexer.CheckConnection(ctx, searchSettings); err != nil {
|
||||
return connectionCheckResponse{Success: false, Message: err.Error()}
|
||||
}
|
||||
return connectionCheckResponse{Success: true, Message: "Meilisearch connection successful"}
|
||||
}
|
||||
|
||||
func (h *AdminHandler) effectiveSettingsForConnectionCheck(
|
||||
ctx context.Context,
|
||||
req adminSettingsConnectionCheckRequest,
|
||||
|
||||
@@ -11,6 +11,7 @@ import (
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/s3client"
|
||||
)
|
||||
@@ -376,6 +377,74 @@ func TestAdminUpdateSettingReportsRestartRequired(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAdminUpdateCatalogSearchSemanticSettingsValidation(t *testing.T) {
|
||||
valid := []struct {
|
||||
key string
|
||||
value string
|
||||
want string
|
||||
}{
|
||||
{key: catalog.SearchSettingMeilisearchSemanticEnabled, value: "TRUE", want: "true"},
|
||||
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: ".30", want: "0.3"},
|
||||
{key: catalog.SearchSettingMeilisearchEmbedder, value: "", want: catalog.DefaultMeilisearchEmbedder},
|
||||
{key: catalog.SearchSettingMeilisearchEmbedder, value: "custom_embedder-2", want: "custom_embedder-2"},
|
||||
}
|
||||
for _, tc := range valid {
|
||||
t.Run("valid "+tc.key, func(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{}
|
||||
handler := &AdminHandler{SettingsRepo: settings, RestartStatus: NewServerRestartStatusTracker()}
|
||||
req := httptest.NewRequest(
|
||||
http.MethodPut,
|
||||
"/admin/settings/"+tc.key,
|
||||
strings.NewReader(`{"value":"`+tc.value+`"}`),
|
||||
)
|
||||
req = withChiParam(req, "key", tc.key)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
handler.HandleUpdateSetting(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if settings.values[tc.key] != tc.want {
|
||||
t.Fatalf("stored value = %q, want %q", settings.values[tc.key], tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
invalid := []struct {
|
||||
key string
|
||||
value string
|
||||
}{
|
||||
{key: catalog.SearchSettingMeilisearchSemanticEnabled, value: "sometimes"},
|
||||
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "-0.01"},
|
||||
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "1.01"},
|
||||
{key: catalog.SearchSettingMeilisearchSemanticRatio, value: "NaN"},
|
||||
{key: catalog.SearchSettingMeilisearchEmbedder, value: "bad.name"},
|
||||
}
|
||||
for _, tc := range invalid {
|
||||
t.Run("invalid "+tc.key, func(t *testing.T) {
|
||||
settings := &fakeServerSettingsStore{}
|
||||
handler := &AdminHandler{SettingsRepo: settings}
|
||||
req := httptest.NewRequest(
|
||||
http.MethodPut,
|
||||
"/admin/settings/"+tc.key,
|
||||
strings.NewReader(`{"value":"`+tc.value+`"}`),
|
||||
)
|
||||
req = withChiParam(req, "key", tc.key)
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
handler.HandleUpdateSetting(rec, req)
|
||||
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400; body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if _, ok := settings.values[tc.key]; ok {
|
||||
t.Fatalf("invalid setting was stored: %#v", settings.values)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func performSettingsCheckRequest(
|
||||
t *testing.T,
|
||||
handler *AdminHandler,
|
||||
|
||||
@@ -18,8 +18,10 @@ type audiobookGroupResponse struct {
|
||||
}
|
||||
|
||||
type audiobookGroupsResponse struct {
|
||||
Total int `json:"total"`
|
||||
Groups []audiobookGroupResponse `json:"groups"`
|
||||
Total int `json:"total"`
|
||||
TotalExact bool `json:"total_exact"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Groups []audiobookGroupResponse `json:"groups"`
|
||||
}
|
||||
|
||||
// HandleGetAudiobookGroups — GET /api/v1/catalog/audiobook-groups
|
||||
@@ -28,7 +30,7 @@ type audiobookGroupsResponse struct {
|
||||
// aggregate stats (book count, total duration, per-profile progress counts)
|
||||
// and up to four poster URLs for cover stacks. Query parameters:
|
||||
// library_id=<id> (required), group_by=author|narrator|series (required),
|
||||
// sort=name|count|duration (default name), limit, offset.
|
||||
// sort=name|count|duration (default name), limit, offset, include_total, q.
|
||||
//
|
||||
// Group names round-trip into the corresponding catalog filter fields
|
||||
// (author / narrator / series), which match case-insensitively.
|
||||
@@ -68,16 +70,27 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
|
||||
limit = n
|
||||
}
|
||||
offset := max(catalog.ParseIntParam(r.URL.Query().Get("offset")), 0)
|
||||
includeTotal := true
|
||||
if raw := strings.TrimSpace(r.URL.Query().Get("include_total")); raw != "" {
|
||||
parsed, parseErr := strconv.ParseBool(raw)
|
||||
if parseErr != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "include_total must be true or false")
|
||||
return
|
||||
}
|
||||
includeTotal = parsed
|
||||
}
|
||||
|
||||
groups, total, err := catalog.ListAudiobookGroups(
|
||||
result, err := catalog.ListAudiobookGroups(
|
||||
r.Context(),
|
||||
h.itemsH.browseRepo.Pool(),
|
||||
catalog.AudiobookGroupsQuery{
|
||||
LibraryID: libraryID,
|
||||
GroupBy: groupBy,
|
||||
Sort: sort,
|
||||
Limit: limit,
|
||||
Offset: offset,
|
||||
LibraryID: libraryID,
|
||||
GroupBy: groupBy,
|
||||
SearchPrefix: strings.TrimSpace(r.URL.Query().Get("q")),
|
||||
IncludeTotal: includeTotal,
|
||||
Sort: sort,
|
||||
Limit: limit,
|
||||
Offset: offset,
|
||||
},
|
||||
h.itemsH.accessFilter(r),
|
||||
)
|
||||
@@ -86,12 +99,18 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
|
||||
return
|
||||
}
|
||||
|
||||
resp := audiobookGroupsResponse{Total: total, Groups: make([]audiobookGroupResponse, 0, len(groups))}
|
||||
for _, g := range groups {
|
||||
resolvedPosters := h.resolveAudiobookGroupPosterURLs(r, result.Groups)
|
||||
resp := audiobookGroupsResponse{
|
||||
Total: result.Total,
|
||||
TotalExact: result.TotalExact,
|
||||
HasMore: result.HasMore,
|
||||
Groups: make([]audiobookGroupResponse, 0, len(result.Groups)),
|
||||
}
|
||||
for _, g := range result.Groups {
|
||||
posterURLs := make([]string, 0, len(g.PosterPaths))
|
||||
for _, path := range g.PosterPaths {
|
||||
if url := h.itemsH.presignURL(r, cardThumbnailPath(path), "card"); url != "" {
|
||||
posterURLs = append(posterURLs, url)
|
||||
if resolved := resolvedPosters[cardThumbnailPath(path)]; resolved.URL != "" {
|
||||
posterURLs = append(posterURLs, resolved.URL)
|
||||
}
|
||||
}
|
||||
resp.Groups = append(resp.Groups, audiobookGroupResponse{
|
||||
@@ -106,3 +125,26 @@ func (h *CatalogHandler) HandleGetAudiobookGroups(w http.ResponseWriter, r *http
|
||||
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) resolveAudiobookGroupPosterURLs(r *http.Request, groups []catalog.AudiobookGroup) map[string]catalog.ResolvedImageURL {
|
||||
if h == nil || h.itemsH == nil || h.itemsH.detailSvc == nil || len(groups) == 0 {
|
||||
return map[string]catalog.ResolvedImageURL{}
|
||||
}
|
||||
|
||||
paths := make([]string, 0, len(groups)*4)
|
||||
seen := make(map[string]struct{}, len(groups)*4)
|
||||
for _, group := range groups {
|
||||
for _, path := range group.PosterPaths {
|
||||
normalized := cardThumbnailPath(path)
|
||||
if normalized == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[normalized]; ok {
|
||||
continue
|
||||
}
|
||||
seen[normalized] = struct{}{}
|
||||
paths = append(paths, normalized)
|
||||
}
|
||||
}
|
||||
return h.itemsH.detailSvc.PresignURLsWithExpiry(r.Context(), paths, "card")
|
||||
}
|
||||
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
@@ -16,9 +17,28 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/sections"
|
||||
)
|
||||
|
||||
// audiobookGroupsCacheTTL matches the client's React Query staleTime: the
|
||||
// grouped author/narrator browse is cached server-side for the same window so
|
||||
// the sequential page fetches and a quick refresh reuse one aggregation.
|
||||
const audiobookGroupsCacheTTL = 60 * time.Second
|
||||
|
||||
type CatalogHandler struct {
|
||||
resolver *catalog.CatalogResolver
|
||||
itemsH *ItemsHandler
|
||||
resolver *catalog.CatalogResolver
|
||||
itemsH *ItemsHandler
|
||||
workSummary catalog.WorkSummaryProvider
|
||||
|
||||
groupsCacheOnce sync.Once
|
||||
groupsCache *catalog.AudiobookGroupsCache
|
||||
}
|
||||
|
||||
// audiobookGroups returns the lazily-initialized grouped-browse cache. Built on
|
||||
// first use (only the route-mounted singleton handler serves audiobook-groups)
|
||||
// so the per-request CatalogHandler instances never spawn a cache sweeper.
|
||||
func (h *CatalogHandler) audiobookGroups() *catalog.AudiobookGroupsCache {
|
||||
h.groupsCacheOnce.Do(func() {
|
||||
h.groupsCache = catalog.NewAudiobookGroupsCache(h.itemsH.browseRepo.Pool(), audiobookGroupsCacheTTL)
|
||||
})
|
||||
return h.groupsCache
|
||||
}
|
||||
|
||||
func NewCatalogHandler(resolver *catalog.CatalogResolver, itemsH *ItemsHandler) *CatalogHandler {
|
||||
@@ -28,12 +48,31 @@ func NewCatalogHandler(resolver *catalog.CatalogResolver, itemsH *ItemsHandler)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) SetWorkSummaryProvider(provider catalog.WorkSummaryProvider) {
|
||||
h.workSummary = provider
|
||||
}
|
||||
|
||||
type catalogResponse struct {
|
||||
Total int `json:"total"`
|
||||
TotalExact bool `json:"total_exact"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Items []itemListResponse `json:"items"`
|
||||
Snapshot string `json:"snapshot,omitempty"`
|
||||
Total int `json:"total"`
|
||||
TotalExact bool `json:"total_exact"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Items []itemListResponse `json:"items"`
|
||||
Snapshot string `json:"snapshot,omitempty"`
|
||||
SearchDiagnostics *searchDiagnostics `json:"search_diagnostics,omitempty"`
|
||||
}
|
||||
|
||||
// searchDiagnostics is an additive, per-query observability object emitted on
|
||||
// /api/v1/catalog only when a relevance-sorted search actually ran through a
|
||||
// CatalogSearchProvider. mode/semantic_used reflect POST-downgrade reality
|
||||
// (a hybrid request that fell back to keyword reports mode="keyword",
|
||||
// semantic_used=false). fallback_reason and index_pending_updates are omitted
|
||||
// when empty.
|
||||
type searchDiagnostics struct {
|
||||
Provider string `json:"provider"`
|
||||
Mode string `json:"mode"`
|
||||
SemanticUsed bool `json:"semantic_used"`
|
||||
FallbackReason string `json:"fallback_reason,omitempty"`
|
||||
IndexPendingUpdates int `json:"index_pending_updates,omitempty"`
|
||||
}
|
||||
|
||||
type catalogFiltersResponse struct {
|
||||
@@ -62,8 +101,36 @@ func (h *CatalogHandler) HandleGetCatalog(w http.ResponseWriter, r *http.Request
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
accessFilter := h.itemsH.accessFilter(r)
|
||||
groupedByWork := strings.EqualFold(strings.TrimSpace(r.URL.Query().Get("group")), "work")
|
||||
if groupedByWork {
|
||||
result, entries, err := h.resolveGroupedCatalogByWork(r, req, accessFilter)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrInvalidCatalogRequest) {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
if errors.Is(err, catalog.ErrCatalogSourceNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Catalog source not found")
|
||||
return
|
||||
}
|
||||
slog.Error("catalog: resolve grouped by work failed", "err_msg", err.Error())
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve catalog")
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.resolver.Resolve(r.Context(), req, h.itemsH.accessFilter(r))
|
||||
resultItems := groupedCatalogItems(entries)
|
||||
items := h.catalogItemResponses(r, resultItems, catalog.NormalizeQuerySort(req.Query.Sort).Field, accessFilter)
|
||||
for i := range items {
|
||||
if i < len(entries) && entries[i].summary != nil {
|
||||
applyWorkSummaryToCatalogItem(&items[i], entries[i].summary)
|
||||
}
|
||||
}
|
||||
h.writeCatalogResponse(w, result, items, groupedByWork)
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.resolver.Resolve(r.Context(), req, accessFilter)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrInvalidCatalogRequest) {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
@@ -77,53 +144,267 @@ func (h *CatalogHandler) HandleGetCatalog(w http.ResponseWriter, r *http.Request
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve catalog")
|
||||
return
|
||||
}
|
||||
items := h.catalogItemResponses(r, result.Items, catalog.NormalizeQuerySort(req.Query.Sort).Field, accessFilter)
|
||||
h.writeCatalogResponse(w, result, items, groupedByWork)
|
||||
}
|
||||
|
||||
overlaySummaries := h.itemsH.listOverlaySummaries(r.Context(), result.Items, h.itemsH.accessFilter(r))
|
||||
userStates := h.itemsH.listItemUserStates(r, result.Items)
|
||||
episodeMetadata := h.itemsH.listEpisodeBrowseMetadata(r.Context(), result.Items)
|
||||
sortField := catalog.NormalizeQuerySort(req.Query.Sort).Field
|
||||
store, profileID, _ := h.itemsH.userStoreForRequest(r)
|
||||
sortMetrics := h.itemsH.listSortMetrics(
|
||||
r.Context(),
|
||||
result.Items,
|
||||
sortField,
|
||||
h.itemsH.accessFilter(r),
|
||||
overlaySummaries,
|
||||
store,
|
||||
apimw.GetUserID(r.Context()),
|
||||
profileID,
|
||||
func (h *CatalogHandler) catalogItemResponses(r *http.Request, resultItems []*models.MediaItem, sortField string, accessFilter catalog.AccessFilter) []itemListResponse {
|
||||
var (
|
||||
localizedItems []*models.MediaItem
|
||||
overlaySummaries map[string]*models.OverlaySummary
|
||||
userStates map[string]*itemUserStateResponse
|
||||
episodeMetadata map[string]struct {
|
||||
SeriesTitle string
|
||||
SeasonNumber *int
|
||||
EpisodeNumber *int
|
||||
}
|
||||
)
|
||||
items := make([]itemListResponse, 0, len(result.Items))
|
||||
for _, item := range result.Items {
|
||||
resp := h.itemsH.toItemListResponseWithOverlay(
|
||||
r,
|
||||
item,
|
||||
overlaySummaries[item.ContentID],
|
||||
userStates[item.ContentID],
|
||||
var enrichWG sync.WaitGroup
|
||||
enrichWG.Add(4)
|
||||
go func() {
|
||||
defer enrichWG.Done()
|
||||
localizedItems = h.itemsH.localizeItemListModels(r.Context(), resultItems, accessFilter)
|
||||
}()
|
||||
go func() {
|
||||
defer enrichWG.Done()
|
||||
overlaySummaries = h.itemsH.listOverlaySummaries(r.Context(), resultItems, accessFilter)
|
||||
}()
|
||||
go func() {
|
||||
defer enrichWG.Done()
|
||||
userStates = h.itemsH.listItemUserStates(r, resultItems)
|
||||
}()
|
||||
go func() {
|
||||
defer enrichWG.Done()
|
||||
episodeMetadata = h.itemsH.listEpisodeBrowseMetadata(r.Context(), resultItems)
|
||||
}()
|
||||
enrichWG.Wait()
|
||||
|
||||
var (
|
||||
imageURLs map[string]itemListImageURLs
|
||||
sortMetrics map[string]*sortMetricsResponse
|
||||
)
|
||||
store, profileID, _ := h.itemsH.userStoreForRequest(r)
|
||||
var responseWG sync.WaitGroup
|
||||
responseWG.Add(2)
|
||||
go func() {
|
||||
defer responseWG.Done()
|
||||
imageURLs = h.itemsH.itemListCardImageURLs(r.Context(), localizedItems)
|
||||
}()
|
||||
go func() {
|
||||
defer responseWG.Done()
|
||||
sortMetrics = h.itemsH.listSortMetrics(
|
||||
r.Context(),
|
||||
resultItems,
|
||||
sortField,
|
||||
accessFilter,
|
||||
overlaySummaries,
|
||||
store,
|
||||
apimw.GetUserID(r.Context()),
|
||||
profileID,
|
||||
)
|
||||
}()
|
||||
responseWG.Wait()
|
||||
|
||||
items := make([]itemListResponse, 0, len(localizedItems))
|
||||
for _, item := range localizedItems {
|
||||
if item == nil {
|
||||
continue
|
||||
}
|
||||
resp := itemListResponseShell(item, overlaySummaries[item.ContentID], userStates[item.ContentID])
|
||||
if meta, ok := episodeMetadata[item.ContentID]; ok {
|
||||
resp.SeriesTitle = meta.SeriesTitle
|
||||
resp.SeasonNumber = meta.SeasonNumber
|
||||
resp.EpisodeNumber = meta.EpisodeNumber
|
||||
}
|
||||
resp.PosterURL = imageURLs[item.ContentID].posterURL
|
||||
resp.BackdropURL = imageURLs[item.ContentID].backdropURL
|
||||
resp.SortMetrics = sortMetrics[item.ContentID]
|
||||
items = append(items, resp)
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) writeCatalogResponse(w http.ResponseWriter, result *catalog.CatalogResult, items []itemListResponse, groupedByWork bool) {
|
||||
var snapshot string
|
||||
if !result.SnapshotAt.IsZero() {
|
||||
snapshot = result.SnapshotAt.Format(time.RFC3339Nano)
|
||||
}
|
||||
|
||||
// A non-empty Provider is the single gate: only the direct-search path sets
|
||||
// it. Browse / preview / non-relevance-sort q= (which never run a provider)
|
||||
// and group=work (fresh CatalogResult with empty Provider) all omit it.
|
||||
var diag *searchDiagnostics
|
||||
if result.Provider != "" {
|
||||
diag = &searchDiagnostics{
|
||||
Provider: result.Provider,
|
||||
Mode: result.Mode,
|
||||
SemanticUsed: result.SemanticUsed,
|
||||
FallbackReason: result.FallbackReason,
|
||||
IndexPendingUpdates: result.IndexPendingEvents,
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, catalogResponse{
|
||||
Total: result.Total,
|
||||
TotalExact: result.TotalExact,
|
||||
HasMore: result.HasMore,
|
||||
Items: items,
|
||||
Snapshot: snapshot,
|
||||
Total: result.Total,
|
||||
TotalExact: result.TotalExact && !groupedByWork,
|
||||
HasMore: result.HasMore,
|
||||
Items: items,
|
||||
Snapshot: snapshot,
|
||||
SearchDiagnostics: diag,
|
||||
})
|
||||
}
|
||||
|
||||
type groupedCatalogEntry struct {
|
||||
item *models.MediaItem
|
||||
summary *catalog.WorkSummary
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) resolveGroupedCatalogByWork(r *http.Request, req catalog.CatalogRequest, accessFilter catalog.AccessFilter) (*catalog.CatalogResult, []groupedCatalogEntry, error) {
|
||||
fetchReq := req
|
||||
fetchReq.Offset = 0
|
||||
fetchReq.Limit = groupedCatalogFetchLimit(req.Limit)
|
||||
fetchReq.SkipTotal = true
|
||||
|
||||
seen := map[string]struct{}{}
|
||||
entries := make([]groupedCatalogEntry, 0, req.Limit+1)
|
||||
groupIndex := 0
|
||||
var snapshot time.Time
|
||||
|
||||
for {
|
||||
result, err := h.resolver.Resolve(r.Context(), fetchReq, accessFilter)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
if snapshot.IsZero() {
|
||||
snapshot = result.SnapshotAt
|
||||
if !snapshot.IsZero() {
|
||||
fetchReq.SnapshotAt = &snapshot
|
||||
}
|
||||
}
|
||||
summaries, err := h.workSummariesForItems(r, result.Items, accessFilter)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
for _, item := range result.Items {
|
||||
summary := summaries[item.ContentID]
|
||||
key := groupedCatalogEntryKey(item, summary)
|
||||
if _, ok := seen[key]; ok {
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
if groupIndex >= req.Offset {
|
||||
entries = append(entries, groupedCatalogEntry{item: item, summary: summary})
|
||||
}
|
||||
groupIndex++
|
||||
if len(entries) > req.Limit {
|
||||
break
|
||||
}
|
||||
}
|
||||
if len(entries) > req.Limit || !result.HasMore || len(result.Items) == 0 {
|
||||
break
|
||||
}
|
||||
fetchReq.Offset += len(result.Items)
|
||||
}
|
||||
|
||||
hasMore := len(entries) > req.Limit
|
||||
if hasMore {
|
||||
entries = entries[:req.Limit]
|
||||
}
|
||||
total := req.Offset + len(entries)
|
||||
if hasMore {
|
||||
total++
|
||||
}
|
||||
result := &catalog.CatalogResult{
|
||||
Items: groupedCatalogItems(entries),
|
||||
Total: total,
|
||||
HasMore: hasMore,
|
||||
TotalExact: false,
|
||||
SnapshotAt: snapshot,
|
||||
}
|
||||
return result, entries, nil
|
||||
}
|
||||
|
||||
func groupedCatalogFetchLimit(limit int) int {
|
||||
switch {
|
||||
case limit <= 0:
|
||||
return 100
|
||||
case limit >= 100:
|
||||
return 100
|
||||
default:
|
||||
return min(100, limit*4+10)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) workSummariesForItems(r *http.Request, items []*models.MediaItem, filter catalog.AccessFilter) (map[string]*catalog.WorkSummary, error) {
|
||||
summaries := map[string]*catalog.WorkSummary{}
|
||||
if h == nil || h.workSummary == nil || len(items) == 0 {
|
||||
return summaries, nil
|
||||
}
|
||||
contentIDs := make([]string, 0, len(items))
|
||||
seen := map[string]struct{}{}
|
||||
for _, item := range items {
|
||||
if !catalogItemCanGroupByWork(item) {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[item.ContentID]; ok {
|
||||
continue
|
||||
}
|
||||
seen[item.ContentID] = struct{}{}
|
||||
contentIDs = append(contentIDs, item.ContentID)
|
||||
}
|
||||
if len(contentIDs) == 0 {
|
||||
return summaries, nil
|
||||
}
|
||||
if batch, ok := h.workSummary.(catalog.WorkSummaryBatchProvider); ok {
|
||||
return batch.ListSummariesForContentIDs(r.Context(), contentIDs, filter)
|
||||
}
|
||||
for _, contentID := range contentIDs {
|
||||
summary, err := h.workSummary.GetSummaryForContentID(r.Context(), contentID, filter)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if summary != nil && summary.WorkID != "" {
|
||||
summaries[contentID] = summary
|
||||
}
|
||||
}
|
||||
return summaries, nil
|
||||
}
|
||||
|
||||
func groupedCatalogItems(entries []groupedCatalogEntry) []*models.MediaItem {
|
||||
items := make([]*models.MediaItem, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
items = append(items, entry.item)
|
||||
}
|
||||
return items
|
||||
}
|
||||
|
||||
func groupedCatalogEntryKey(item *models.MediaItem, summary *catalog.WorkSummary) string {
|
||||
if catalogItemCanGroupByWork(item) && summary != nil && summary.WorkID != "" {
|
||||
return "work:" + summary.WorkID
|
||||
}
|
||||
if item == nil {
|
||||
return "item:"
|
||||
}
|
||||
return "item:" + item.ContentID
|
||||
}
|
||||
|
||||
func catalogItemCanGroupByWork(item *models.MediaItem) bool {
|
||||
return item != nil && (item.Type == "ebook" || item.Type == "audiobook")
|
||||
}
|
||||
|
||||
func applyWorkSummaryToCatalogItem(item *itemListResponse, summary *catalog.WorkSummary) {
|
||||
if item == nil || summary == nil || summary.WorkID == "" {
|
||||
return
|
||||
}
|
||||
item.Type = "work"
|
||||
item.WorkID = summary.WorkID
|
||||
item.WorkTitle = summary.Title
|
||||
item.WorkFormats = summary.Formats
|
||||
if summary.Title != "" {
|
||||
item.Title = summary.Title
|
||||
}
|
||||
}
|
||||
|
||||
func (h *CatalogHandler) HandleGetCatalogFilters(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.resolver == nil || h.itemsH == nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Catalog is not configured")
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
)
|
||||
|
||||
// decodeCatalogResponse runs writeCatalogResponse against an httptest recorder
|
||||
// and returns the decoded JSON body as a generic map so individual keys can be
|
||||
// asserted for presence/absence (search_diagnostics is omitempty).
|
||||
func decodeCatalogResponse(t *testing.T, result *catalog.CatalogResult, grouped bool) map[string]any {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
// writeCatalogResponse uses no handler state, so a zero-value handler is fine.
|
||||
(&CatalogHandler{}).writeCatalogResponse(rec, result, []itemListResponse{}, grouped)
|
||||
|
||||
var body map[string]any
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decoding catalog response: %v\nbody: %s", err, rec.Body.String())
|
||||
}
|
||||
return body
|
||||
}
|
||||
|
||||
func TestWriteCatalogResponse_DiagnosticsKeywordFallback(t *testing.T) {
|
||||
body := decodeCatalogResponse(t, &catalog.CatalogResult{
|
||||
Total: 3,
|
||||
TotalExact: true,
|
||||
HasMore: false,
|
||||
Provider: catalog.SearchProviderMeilisearch,
|
||||
Mode: "keyword",
|
||||
SemanticUsed: false,
|
||||
FallbackReason: `semantic_not_ready: type "movie" coverage 40% below threshold`,
|
||||
}, false)
|
||||
|
||||
// Existing keys must stay present and unchanged (byte-stable contract).
|
||||
for _, key := range []string{"total", "total_exact", "has_more", "items"} {
|
||||
if _, ok := body[key]; !ok {
|
||||
t.Fatalf("response missing existing key %q: %v", key, body)
|
||||
}
|
||||
}
|
||||
if body["total"].(float64) != 3 {
|
||||
t.Fatalf("total = %v, want 3", body["total"])
|
||||
}
|
||||
if body["total_exact"].(bool) != true {
|
||||
t.Fatalf("total_exact = %v, want true", body["total_exact"])
|
||||
}
|
||||
|
||||
diagRaw, ok := body["search_diagnostics"]
|
||||
if !ok {
|
||||
t.Fatalf("expected search_diagnostics in response: %v", body)
|
||||
}
|
||||
diag := diagRaw.(map[string]any)
|
||||
if diag["provider"] != catalog.SearchProviderMeilisearch {
|
||||
t.Fatalf("provider = %v, want %q", diag["provider"], catalog.SearchProviderMeilisearch)
|
||||
}
|
||||
if diag["mode"] != "keyword" {
|
||||
t.Fatalf("mode = %v, want keyword", diag["mode"])
|
||||
}
|
||||
if diag["semantic_used"].(bool) != false {
|
||||
t.Fatalf("semantic_used = %v, want false", diag["semantic_used"])
|
||||
}
|
||||
if diag["fallback_reason"] != `semantic_not_ready: type "movie" coverage 40% below threshold` {
|
||||
t.Fatalf("fallback_reason = %v", diag["fallback_reason"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteCatalogResponse_DiagnosticsHybridOmitsFallbackReason(t *testing.T) {
|
||||
body := decodeCatalogResponse(t, &catalog.CatalogResult{
|
||||
Provider: catalog.SearchProviderMeilisearch,
|
||||
Mode: "hybrid",
|
||||
SemanticUsed: true,
|
||||
IndexPendingEvents: 7,
|
||||
}, false)
|
||||
|
||||
diagRaw, ok := body["search_diagnostics"]
|
||||
if !ok {
|
||||
t.Fatalf("expected search_diagnostics in response: %v", body)
|
||||
}
|
||||
diag := diagRaw.(map[string]any)
|
||||
if diag["semantic_used"].(bool) != true {
|
||||
t.Fatalf("semantic_used = %v, want true", diag["semantic_used"])
|
||||
}
|
||||
if diag["mode"] != "hybrid" {
|
||||
t.Fatalf("mode = %v, want hybrid", diag["mode"])
|
||||
}
|
||||
if _, ok := diag["fallback_reason"]; ok {
|
||||
t.Fatalf("fallback_reason should be omitted when empty: %v", diag)
|
||||
}
|
||||
if diag["index_pending_updates"].(float64) != 7 {
|
||||
t.Fatalf("index_pending_updates = %v, want 7", diag["index_pending_updates"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteCatalogResponse_NoProviderOmitsDiagnostics(t *testing.T) {
|
||||
// Browse / preview / non-relevance q= paths never set Provider.
|
||||
body := decodeCatalogResponse(t, &catalog.CatalogResult{
|
||||
Total: 5,
|
||||
TotalExact: true,
|
||||
}, false)
|
||||
|
||||
if _, ok := body["search_diagnostics"]; ok {
|
||||
t.Fatalf("search_diagnostics should be omitted when no provider search ran: %v", body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteCatalogResponse_GroupedByWorkOmitsDiagnostics(t *testing.T) {
|
||||
// group=work builds a fresh CatalogResult with an empty Provider.
|
||||
body := decodeCatalogResponse(t, &catalog.CatalogResult{
|
||||
Total: 2,
|
||||
TotalExact: true,
|
||||
}, true)
|
||||
|
||||
if _, ok := body["search_diagnostics"]; ok {
|
||||
t.Fatalf("grouped response should omit search_diagnostics: %v", body)
|
||||
}
|
||||
// Grouped responses force total_exact false regardless of result.TotalExact.
|
||||
if body["total_exact"].(bool) != false {
|
||||
t.Fatalf("grouped total_exact = %v, want false", body["total_exact"])
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeCatalogWorkSummaryProvider struct{}
|
||||
|
||||
func (fakeCatalogWorkSummaryProvider) GetSummaryForContentID(ctx context.Context, contentID string, filter catalog.AccessFilter) (*catalog.WorkSummary, error) {
|
||||
if contentID != "ebook-1" && contentID != "audio-1" {
|
||||
return nil, nil
|
||||
}
|
||||
return &catalog.WorkSummary{
|
||||
WorkID: "work-1",
|
||||
Title: "Project Hail Mary",
|
||||
Formats: []catalog.WorkFormatSummary{
|
||||
{Type: "ebook", ContentID: "ebook-1", LibraryID: 1},
|
||||
{Type: "audiobook", ContentID: "audio-1", LibraryID: 2},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestGroupedCatalogEntryKeyDeduplicatesLinkedFormats(t *testing.T) {
|
||||
summary := &catalog.WorkSummary{
|
||||
WorkID: "work-1",
|
||||
Title: "Project Hail Mary",
|
||||
Formats: []catalog.WorkFormatSummary{
|
||||
{Type: "ebook", ContentID: "ebook-1", LibraryID: 1},
|
||||
{Type: "audiobook", ContentID: "audio-1", LibraryID: 2},
|
||||
},
|
||||
}
|
||||
|
||||
ebookKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "ebook-1", Type: "ebook"}, summary)
|
||||
audioKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "audio-1", Type: "audiobook"}, summary)
|
||||
|
||||
if ebookKey != audioKey {
|
||||
t.Fatalf("linked format keys differ: %q vs %q", ebookKey, audioKey)
|
||||
}
|
||||
movieKey := groupedCatalogEntryKey(&models.MediaItem{ContentID: "movie-1", Type: "movie"}, nil)
|
||||
if movieKey != "item:movie-1" {
|
||||
t.Fatalf("movie key = %q, want item key", movieKey)
|
||||
}
|
||||
|
||||
resp := itemListResponse{ContentID: "ebook-1", Type: "ebook", Title: "Project Hail Mary"}
|
||||
applyWorkSummaryToCatalogItem(&resp, summary)
|
||||
if resp.Type != "work" || resp.WorkID != "work-1" || len(resp.WorkFormats) != 2 {
|
||||
t.Fatalf("work response = %#v", resp)
|
||||
}
|
||||
}
|
||||
@@ -87,6 +87,37 @@ func (h *CatalogResourceHandler) HandleGetItemVersions(w http.ResponseWriter, r
|
||||
writeJSON(w, http.StatusOK, detail.Versions)
|
||||
}
|
||||
|
||||
// HandleGetMangaFiles returns the local file listing for a manga series (the
|
||||
// series "View Details" dialog): folder paths plus per-chapter file rows.
|
||||
// Folder and file paths are stripped for viewers without file-path visibility,
|
||||
// matching the item-versions policy; file names and sizes remain.
|
||||
func (h *CatalogResourceHandler) HandleGetMangaFiles(w http.ResponseWriter, r *http.Request) {
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Item ID is required")
|
||||
return
|
||||
}
|
||||
|
||||
files, err := h.items.detailSvc.GetMangaChapterFiles(r.Context(), id, h.items.accessFilter(r))
|
||||
if err != nil {
|
||||
if isNotFound(err) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get manga files")
|
||||
return
|
||||
}
|
||||
|
||||
if !h.items.requestCanViewFilePaths(r) {
|
||||
files.FolderPaths = nil
|
||||
for i := range files.Files {
|
||||
files.Files[i].FilePath = ""
|
||||
}
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, files)
|
||||
}
|
||||
|
||||
func (h *CatalogResourceHandler) HandleGetItemEpisodes(w http.ResponseWriter, r *http.Request) {
|
||||
filter := h.items.accessFilter(r)
|
||||
id := chi.URLParam(r, "id")
|
||||
@@ -197,12 +228,24 @@ func (h *CatalogResourceHandler) HandleGetSeasons(w http.ResponseWriter, r *http
|
||||
}
|
||||
|
||||
if len(seasons) > 0 {
|
||||
episodesBySeason, err := h.items.episodeRepo.ListBySeriesGroupedBySeason(r.Context(), id)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list seasons")
|
||||
return
|
||||
}
|
||||
progressMap, hasProgressMap := h.items.progressMapForEpisodes(r, flattenEpisodeGroups(episodesBySeason))
|
||||
|
||||
resp := make([]seasonResponse, 0, len(seasons))
|
||||
for _, s := range seasons {
|
||||
sr := h.items.toSeasonResponse(r, id, s)
|
||||
if sr.EpisodeCount == 0 {
|
||||
episodes := episodesBySeason[s.SeasonNumber]
|
||||
if len(episodes) == 0 {
|
||||
continue
|
||||
}
|
||||
var userData *catalog.SeasonUserData
|
||||
if hasProgressMap {
|
||||
userData = catalog.EpisodeRollupUserData(episodes, progressMap)
|
||||
}
|
||||
sr := h.items.toSeasonResponseFromEpisodes(r, id, s, episodes, userData)
|
||||
resp = append(resp, sr)
|
||||
}
|
||||
|
||||
@@ -224,13 +267,14 @@ func (h *CatalogResourceHandler) HandleGetSeasons(w http.ResponseWriter, r *http
|
||||
title = "Specials"
|
||||
}
|
||||
episodes, _ := h.items.episodeRepo.ListBySeason(r.Context(), id, s.SeasonNumber)
|
||||
userData := h.items.getAggregateUserData(r, episodes)
|
||||
resp = append(resp, seasonResponse{
|
||||
ContentID: fmt.Sprintf("%s-S%02d", id, s.SeasonNumber),
|
||||
SeasonNumber: s.SeasonNumber,
|
||||
IsSpecials: s.SeasonNumber == 0,
|
||||
EpisodeCount: s.EpisodeCount,
|
||||
Title: title,
|
||||
UserData: h.items.getAggregateUserData(r, episodes),
|
||||
UserData: userData,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -273,10 +317,27 @@ func (h *CatalogResourceHandler) HandleGetSeason(w http.ResponseWriter, r *http.
|
||||
season, err := h.items.seasonRepo.GetBySeriesAndNumber(r.Context(), id, num)
|
||||
switch {
|
||||
case err == nil:
|
||||
episodes, _ := h.items.episodeRepo.ListBySeasonID(r.Context(), season.ContentID)
|
||||
episodes, err := h.items.episodeRepo.ListBySeasonID(r.Context(), season.ContentID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get season")
|
||||
return
|
||||
}
|
||||
if len(episodes) == 0 {
|
||||
episodes, err = h.items.episodeRepo.ListBySeason(r.Context(), id, season.SeasonNumber)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to get season")
|
||||
return
|
||||
}
|
||||
}
|
||||
h.items.maybeRequestStaleSeasonMetadataRefresh(r.Context(), season.ContentID, episodes)
|
||||
writeJSON(w, http.StatusOK, seasonDetailResponse{
|
||||
Season: h.items.toSeasonResponse(r, id, season),
|
||||
Season: h.items.toSeasonResponseFromEpisodes(
|
||||
r,
|
||||
id,
|
||||
season,
|
||||
episodes,
|
||||
h.items.getAggregateUserData(r, episodes),
|
||||
),
|
||||
})
|
||||
return
|
||||
case !errors.Is(err, catalog.ErrSeasonNotFound):
|
||||
@@ -388,7 +449,13 @@ func (h *CatalogResourceHandler) syntheticSeasonDetail(r *http.Request, seasonID
|
||||
} else {
|
||||
season.Title = "Season " + strconv.Itoa(seasonNum)
|
||||
}
|
||||
seasonResp := h.items.toSeasonResponse(r, seriesID, season)
|
||||
seasonResp := h.items.toSeasonResponseFromEpisodes(
|
||||
r,
|
||||
seriesID,
|
||||
season,
|
||||
episodes,
|
||||
h.items.getAggregateUserData(r, episodes),
|
||||
)
|
||||
return &catalog.ItemDetail{
|
||||
ContentID: seasonID,
|
||||
Type: "season",
|
||||
|
||||
@@ -43,6 +43,7 @@ type createCollectionRequest struct {
|
||||
AllowedProfileIDs []string `json:"allowed_profile_ids"`
|
||||
QueryDefinition json.RawMessage `json:"query_definition"`
|
||||
SortConfig json.RawMessage `json:"sort_config"`
|
||||
DisplayQueryDefinition json.RawMessage `json:"display_query_definition"`
|
||||
IncludeInServerCollections bool `json:"include_in_server_collections"`
|
||||
PosterSourceURL string `json:"poster_source_url"`
|
||||
}
|
||||
@@ -56,6 +57,8 @@ type updateCollectionRequest struct {
|
||||
SortConfig json.RawMessage `json:"sort_config"`
|
||||
SourceURL *string `json:"source_url"`
|
||||
MaxItems *int `json:"max_items"`
|
||||
LibraryIDs *[]int `json:"library_ids"`
|
||||
DisplayQueryDefinition json.RawMessage `json:"display_query_definition"`
|
||||
IncludeInServerCollections *bool `json:"include_in_server_collections"`
|
||||
PosterSourceURL *string `json:"poster_source_url"`
|
||||
GroupID optionalNullableString `json:"group_id"`
|
||||
@@ -85,6 +88,7 @@ type collectionResponse struct {
|
||||
LastSyncAt string `json:"last_sync_at,omitempty"`
|
||||
LastSyncStatus string `json:"last_sync_status,omitempty"`
|
||||
LastSyncMessage string `json:"last_sync_message,omitempty"`
|
||||
DisplayQueryDefinition json.RawMessage `json:"display_query_definition,omitempty"`
|
||||
ItemCount int `json:"item_count"`
|
||||
IncludeInServerCollections bool `json:"include_in_server_collections"`
|
||||
PosterURL string `json:"poster_url,omitempty"`
|
||||
@@ -98,6 +102,19 @@ type collectionListResponse struct {
|
||||
Groups []collectionGroupResponse `json:"groups"`
|
||||
}
|
||||
|
||||
type collectionCapabilitiesResponse struct {
|
||||
// DisplayFilterFields are the catalog query fields a personal-collection
|
||||
// display filter may use. Clients build a display_query_definition fragment
|
||||
// from these rather than a bespoke enum.
|
||||
DisplayFilterFields []string `json:"display_filter_fields"`
|
||||
DisplayFilterPresets collectionDisplayFilterPresets `json:"display_filter_presets"`
|
||||
}
|
||||
|
||||
type collectionDisplayFilterPresets struct {
|
||||
Watched []string `json:"watched"`
|
||||
Media []string `json:"media"`
|
||||
}
|
||||
|
||||
type collectionGroupResponse struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
@@ -208,6 +225,17 @@ func (h *CollectionHandler) HandleListCollections(w http.ResponseWriter, r *http
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
// HandleCapabilities exposes additive feature support for collection clients.
|
||||
func (h *CollectionHandler) HandleCapabilities(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, collectionCapabilitiesResponse{
|
||||
DisplayFilterFields: []string{"type", "watched"},
|
||||
DisplayFilterPresets: collectionDisplayFilterPresets{
|
||||
Watched: []string{"all", "watched", "unwatched"},
|
||||
Media: []string{"all", "movie", "series"},
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
// HandleCreateCollection handles POST /collections.
|
||||
func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
@@ -247,6 +275,11 @@ func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *htt
|
||||
}
|
||||
queryDefinition := string(queryDefinitionJSON)
|
||||
sortConfig := string(defaultJSON(req.SortConfig))
|
||||
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(req.DisplayQueryDefinition)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
collection, err := store.CreateCollection(r.Context(), userstore.CreateCollectionInput{
|
||||
CreatorProfileID: profileID,
|
||||
Name: req.Name,
|
||||
@@ -255,6 +288,7 @@ func (h *CollectionHandler) HandleCreateCollection(w http.ResponseWriter, r *htt
|
||||
AllowedProfileIDs: req.AllowedProfileIDs,
|
||||
QueryDefinition: queryDefinition,
|
||||
SortConfig: sortConfig,
|
||||
DisplayQueryDefinition: displayQueryDefinition,
|
||||
IncludeInServerCollections: req.IncludeInServerCollections,
|
||||
})
|
||||
if err != nil {
|
||||
@@ -306,6 +340,14 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
|
||||
AllowedProfileIDs: req.AllowedProfileIDs,
|
||||
IncludeInServerCollections: req.IncludeInServerCollections,
|
||||
}
|
||||
if req.DisplayQueryDefinition != nil {
|
||||
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(req.DisplayQueryDefinition)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
input.DisplayQueryDefinition = &displayQueryDefinition
|
||||
}
|
||||
if len(req.QueryDefinition) > 0 {
|
||||
normalized, err := normalizeSmartCollectionQueryDefinitionJSON(req.QueryDefinition, true, true)
|
||||
if err != nil {
|
||||
@@ -341,7 +383,7 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
|
||||
// Source URL and max items both live inside source_config (Limit / URL).
|
||||
// Load the existing collection and re-marshal so the unaffected fields
|
||||
// (preset, media_type, etc.) survive untouched.
|
||||
if req.SourceURL != nil || req.MaxItems != nil {
|
||||
if req.SourceURL != nil || req.MaxItems != nil || req.LibraryIDs != nil {
|
||||
existing, err := store.GetCollection(r.Context(), collectionID)
|
||||
if err != nil {
|
||||
if strings.Contains(err.Error(), "not found") {
|
||||
@@ -356,6 +398,18 @@ func (h *CollectionHandler) HandleUpdateCollection(w http.ResponseWriter, r *htt
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to parse source config")
|
||||
return
|
||||
}
|
||||
if req.LibraryIDs != nil {
|
||||
if !catalog.IsSyncableType(existing.CollectionType) {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "library_ids can only be edited for imported collections")
|
||||
return
|
||||
}
|
||||
if !validateOptionalLibraryIDs(*req.LibraryIDs, w) {
|
||||
return
|
||||
}
|
||||
cfg.LibraryIDs = append([]int(nil), (*req.LibraryIDs)...)
|
||||
emptyQuery := "{}"
|
||||
input.QueryDefinition = &emptyQuery
|
||||
}
|
||||
if req.MaxItems != nil {
|
||||
if *req.MaxItems < 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "max_items must be zero or positive")
|
||||
@@ -805,6 +859,9 @@ func toCollectionResponse(c userstore.Collection) collectionResponse {
|
||||
CreatedAt: c.CreatedAt,
|
||||
UpdatedAt: c.UpdatedAt,
|
||||
}
|
||||
if strings.TrimSpace(c.DisplayQueryDefinition) != "" {
|
||||
resp.DisplayQueryDefinition = json.RawMessage(c.DisplayQueryDefinition)
|
||||
}
|
||||
if c.SourceConfig != "" && c.SourceConfig != "{}" {
|
||||
resp.SourceConfig = json.RawMessage(c.SourceConfig)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
)
|
||||
|
||||
// subscriptionRequest is the JSON body for POST /downloads/subscriptions.
|
||||
type subscriptionRequest struct {
|
||||
SeriesID string `json:"series_id"`
|
||||
Mode string `json:"mode"` // all | future | latest_season | specific_seasons
|
||||
SeasonNumbers []int `json:"season_numbers,omitempty"`
|
||||
DeleteWatched bool `json:"delete_watched,omitempty"`
|
||||
MaxStorageBytes int64 `json:"max_storage_bytes,omitempty"`
|
||||
}
|
||||
|
||||
// patchSubscriptionRequest is the JSON body for PATCH /downloads/subscriptions/{id}.
|
||||
// Pointer fields distinguish "absent" from a zero value so a partial update only
|
||||
// touches the fields the client sent.
|
||||
type patchSubscriptionRequest struct {
|
||||
Mode *string `json:"mode,omitempty"`
|
||||
SeasonNumbers *[]int `json:"season_numbers,omitempty"`
|
||||
DeleteWatched *bool `json:"delete_watched,omitempty"`
|
||||
MaxStorageBytes *int64 `json:"max_storage_bytes,omitempty"`
|
||||
Active *bool `json:"active,omitempty"`
|
||||
}
|
||||
|
||||
// subscriptionResponse is one subscription in API responses.
|
||||
type subscriptionResponse struct {
|
||||
ID string `json:"id"`
|
||||
SeriesID string `json:"series_id"`
|
||||
Mode string `json:"mode"`
|
||||
SeasonNumbers []int `json:"season_numbers,omitempty"`
|
||||
TargetSeason *int `json:"target_season,omitempty"`
|
||||
DeleteWatched bool `json:"delete_watched"`
|
||||
MaxStorageBytes int64 `json:"max_storage_bytes"`
|
||||
Active bool `json:"active"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
type subscriptionsListResponse struct {
|
||||
Subscriptions []subscriptionResponse `json:"subscriptions"`
|
||||
}
|
||||
|
||||
// subscriptionResultResponse wraps a created/updated subscription plus the count
|
||||
// of in-scope episodes registered as managed downloads.
|
||||
type subscriptionResultResponse struct {
|
||||
Subscription subscriptionResponse `json:"subscription"`
|
||||
Registered int `json:"registered"`
|
||||
}
|
||||
|
||||
// subscriptionSyncResponse reports how many newly in-scope episodes a sync
|
||||
// registered across the calling device's monitors.
|
||||
type subscriptionSyncResponse struct {
|
||||
Registered int `json:"registered"`
|
||||
}
|
||||
|
||||
func toSubscriptionResponse(s *downloads.Subscription) subscriptionResponse {
|
||||
return subscriptionResponse{
|
||||
ID: s.ID,
|
||||
SeriesID: s.SeriesID,
|
||||
Mode: s.Mode,
|
||||
SeasonNumbers: s.SeasonNumbers,
|
||||
TargetSeason: s.TargetSeason,
|
||||
DeleteWatched: s.DeleteWatched,
|
||||
MaxStorageBytes: s.MaxStorageBytes,
|
||||
Active: s.Active,
|
||||
CreatedAt: s.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
UpdatedAt: s.UpdatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
}
|
||||
}
|
||||
|
||||
func toSubscriptionResultResponse(result *downloads.SubscriptionResult) subscriptionResultResponse {
|
||||
return subscriptionResultResponse{
|
||||
Subscription: toSubscriptionResponse(result.Subscription),
|
||||
Registered: result.Registered,
|
||||
}
|
||||
}
|
||||
|
||||
// HandleCreateSubscription handles POST /downloads/subscriptions (managed-only).
|
||||
func (h *DownloadHandler) HandleCreateSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, deviceName, devicePlatform, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req subscriptionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
if req.SeriesID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "series_id is required")
|
||||
return
|
||||
}
|
||||
|
||||
result, err := h.svc.CreateSubscription(r.Context(), userID, downloads.SubscriptionRequest{
|
||||
SeriesID: req.SeriesID,
|
||||
Mode: req.Mode,
|
||||
SeasonNumbers: req.SeasonNumbers,
|
||||
DeleteWatched: req.DeleteWatched,
|
||||
MaxStorageBytes: req.MaxStorageBytes,
|
||||
ProfileID: profileID,
|
||||
DeviceID: deviceID,
|
||||
DeviceName: deviceName,
|
||||
DevicePlatform: devicePlatform,
|
||||
}, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, toSubscriptionResultResponse(result))
|
||||
}
|
||||
|
||||
// HandleListSubscriptions handles GET /downloads/subscriptions (managed-only).
|
||||
func (h *DownloadHandler) HandleListSubscriptions(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
subs, err := h.svc.ListSubscriptions(r.Context(), userID, profileID, deviceID)
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
responses := make([]subscriptionResponse, 0, len(subs))
|
||||
for _, s := range subs {
|
||||
responses = append(responses, toSubscriptionResponse(s))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, subscriptionsListResponse{Subscriptions: responses})
|
||||
}
|
||||
|
||||
// HandleGetSubscription handles GET /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandleGetSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
sub, err := h.svc.GetSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id"))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, toSubscriptionResponse(sub))
|
||||
}
|
||||
|
||||
// HandlePatchSubscription handles PATCH /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandlePatchSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
var req patchSubscriptionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
result, err := h.svc.UpdateSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id"), downloads.SubscriptionPatch{
|
||||
Mode: req.Mode,
|
||||
SeasonNumbers: req.SeasonNumbers,
|
||||
DeleteWatched: req.DeleteWatched,
|
||||
MaxStorageBytes: req.MaxStorageBytes,
|
||||
Active: req.Active,
|
||||
}, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, toSubscriptionResultResponse(result))
|
||||
}
|
||||
|
||||
// HandleDeleteSubscription handles DELETE /downloads/subscriptions/{id} (managed-only).
|
||||
func (h *DownloadHandler) HandleDeleteSubscription(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := h.svc.DeleteSubscription(r.Context(), userID, profileID, deviceID, chi.URLParam(r, "id")); err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// HandleSyncSubscriptions handles POST /downloads/subscriptions/sync (managed-only):
|
||||
// the client triggers it on open / background refresh to register newly available
|
||||
// episodes for its monitors, then pulls the files on its own schedule.
|
||||
func (h *DownloadHandler) HandleSyncSubscriptions(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
registered, err := h.svc.SyncSubscriptions(r.Context(), userID, profileID, deviceID, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeSubscriptionError(w, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, subscriptionSyncResponse{Registered: registered})
|
||||
}
|
||||
|
||||
// writeSubscriptionError maps series-monitoring errors to HTTP responses.
|
||||
func (h *DownloadHandler) writeSubscriptionError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrSubscriptionNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Subscription not found")
|
||||
case errors.Is(err, downloads.ErrSubscriptionsUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Series monitoring is not available")
|
||||
case errors.Is(err, downloads.ErrInvalidSubscriptionMode):
|
||||
writeError(w, http.StatusBadRequest, "invalid_mode", "Unknown monitor mode")
|
||||
case errors.Is(err, downloads.ErrSeasonsRequired):
|
||||
writeError(w, http.StatusBadRequest, "seasons_required", "season_numbers is required for specific_seasons mode")
|
||||
case errors.Is(err, downloads.ErrInvalidSeasonNumbers):
|
||||
writeError(w, http.StatusBadRequest, "invalid_season_numbers", "season_numbers values must be between 0 and 9999")
|
||||
case errors.Is(err, downloads.ErrNotSeries):
|
||||
writeError(w, http.StatusBadRequest, "not_series", "content_id is not a series")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile and device are required for series monitoring")
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled), errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
case errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Series not found")
|
||||
default:
|
||||
slog.Error("download subscription operation failed", "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to process subscription")
|
||||
}
|
||||
}
|
||||
@@ -12,17 +12,33 @@ import (
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/download"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
|
||||
// DownloadService is the interface that the download handler depends on.
|
||||
// DownloadService is the interface that the download handler depends on. A
|
||||
// non-empty deviceID (from the X-Silo-Device-Id header) selects the managed
|
||||
// device-library lifecycle; empty is the ephemeral/account-level path.
|
||||
type DownloadService interface {
|
||||
CreateQueued(ctx context.Context, userID int, req download.CreateRequest, filter catalog.AccessFilter) (*download.Download, error)
|
||||
CreateQueuedBatch(ctx context.Context, userID int, seriesContentID string, filter catalog.AccessFilter) ([]*download.Download, string, error)
|
||||
ServeDirect(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, fileID int, filter catalog.AccessFilter) error
|
||||
ServeFile(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, downloadID string) error
|
||||
List(ctx context.Context, userID int) ([]*download.Download, error)
|
||||
Cancel(ctx context.Context, userID int, downloadID string) error
|
||||
Capability(ctx context.Context, userID int) (downloads.Capability, error)
|
||||
Create(ctx context.Context, userID int, req downloads.CreateRequest, filter catalog.AccessFilter) (*downloads.Download, error)
|
||||
CreateSeries(ctx context.Context, userID int, req downloads.CreateRequest, filter catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error)
|
||||
CreateSeason(ctx context.Context, userID int, req downloads.CreateRequest, seasonNumber int, filter catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error)
|
||||
CreateSubscription(ctx context.Context, userID int, req downloads.SubscriptionRequest, filter catalog.AccessFilter) (*downloads.SubscriptionResult, error)
|
||||
ListSubscriptions(ctx context.Context, userID int, profileID, deviceID string) ([]*downloads.Subscription, error)
|
||||
GetSubscription(ctx context.Context, userID int, profileID, deviceID, id string) (*downloads.Subscription, error)
|
||||
UpdateSubscription(ctx context.Context, userID int, profileID, deviceID, id string, patch downloads.SubscriptionPatch, filter catalog.AccessFilter) (*downloads.SubscriptionResult, error)
|
||||
DeleteSubscription(ctx context.Context, userID int, profileID, deviceID, id string) error
|
||||
SyncSubscriptions(ctx context.Context, userID int, profileID, deviceID string, filter catalog.AccessFilter) (int, error)
|
||||
ServeDirect(ctx context.Context, w http.ResponseWriter, r *http.Request, userID, fileID int, format string, filter catalog.AccessFilter) error
|
||||
ServeFile(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID string, filter catalog.AccessFilter) error
|
||||
List(ctx context.Context, userID int, profileID, deviceID string) ([]*downloads.Download, error)
|
||||
Delete(ctx context.Context, userID int, profileID, deviceID, downloadID string) error
|
||||
PatchStatus(ctx context.Context, userID int, profileID, deviceID, downloadID, status string) error
|
||||
BuildManifest(ctx context.Context, userID int, profileID, deviceID, downloadID string, filter catalog.AccessFilter) (*downloads.OfflineManifest, error)
|
||||
BuildBatchManifests(ctx context.Context, userID int, profileID, deviceID, batchID string, filter catalog.AccessFilter) ([]*downloads.OfflineManifest, []downloads.SkippedManifest, error)
|
||||
ServeArtwork(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID, kind string, filter catalog.AccessFilter) error
|
||||
ServeSubtitle(ctx context.Context, w http.ResponseWriter, r *http.Request, userID int, profileID, deviceID, downloadID, ref string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
// DownloadHandler handles download endpoints.
|
||||
@@ -37,44 +53,96 @@ func NewDownloadHandler(svc DownloadService) *DownloadHandler {
|
||||
|
||||
// downloadRequest represents the JSON body for POST /downloads.
|
||||
type downloadRequest struct {
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
FileID int `json:"file_id,omitempty"`
|
||||
Series bool `json:"series,omitempty"` // if true, downloads all episodes
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
FileID int `json:"file_id,omitempty"`
|
||||
Quality string `json:"quality,omitempty"` // original (default) | 20mbps | 10mbps | 5mbps | 2mbps | 1mbps
|
||||
Series bool `json:"series,omitempty"` // if true, downloads all episodes
|
||||
Season *int `json:"season_number,omitempty"` // with series=true, downloads only this season (0 = Specials)
|
||||
Caps *downloadCaps `json:"caps,omitempty"` // device decode capability (original fallback / transcode target)
|
||||
}
|
||||
|
||||
// downloadCaps mirrors playback.ClientCapabilities for the request body.
|
||||
type downloadCaps struct {
|
||||
CodecsVideo []string `json:"codecs_video,omitempty"`
|
||||
CodecsAudio []string `json:"codecs_audio,omitempty"`
|
||||
AudioPassthroughCodecs []string `json:"audio_passthrough_codecs,omitempty"`
|
||||
Containers []string `json:"containers,omitempty"`
|
||||
MaxResolution string `json:"max_resolution,omitempty"`
|
||||
HDR bool `json:"hdr,omitempty"`
|
||||
}
|
||||
|
||||
// patchDownloadRequest is the JSON body for PATCH /downloads/{id}.
|
||||
type patchDownloadRequest struct {
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
// downloadResponse represents a download entry in API responses.
|
||||
type downloadResponse struct {
|
||||
ID string `json:"id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
BatchID string `json:"batch_id,omitempty"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
BytesSent int64 `json:"bytes_sent"`
|
||||
Kind string `json:"kind"`
|
||||
Status string `json:"status"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
CompletedAt *string `json:"completed_at,omitempty"`
|
||||
ID string `json:"id"`
|
||||
ContentID string `json:"content_id"`
|
||||
EpisodeID string `json:"episode_id,omitempty"`
|
||||
BatchID string `json:"batch_id,omitempty"`
|
||||
DeviceID string `json:"device_id,omitempty"`
|
||||
MediaFileID int `json:"media_file_id"`
|
||||
FileSize int64 `json:"file_size"`
|
||||
BytesSent int64 `json:"bytes_sent"`
|
||||
Kind string `json:"kind"`
|
||||
Status string `json:"status"`
|
||||
Quality string `json:"quality"`
|
||||
EffectiveQuality string `json:"effective_quality"`
|
||||
DeliveryFormat string `json:"delivery_format"`
|
||||
TargetBitrateKbps int `json:"target_bitrate_kbps"`
|
||||
Revision int `json:"revision"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
CompletedAt *string `json:"completed_at,omitempty"`
|
||||
}
|
||||
|
||||
// downloadsListResponse wraps the downloads list for JSON serialization.
|
||||
type downloadsListResponse struct {
|
||||
Downloads []downloadResponse `json:"downloads"`
|
||||
Downloads []downloadResponse `json:"downloads"`
|
||||
Skipped []downloads.SkippedDownload `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
func toDownloadResponse(d *download.Download) downloadResponse {
|
||||
type batchManifestsResponse struct {
|
||||
Manifests []*downloads.OfflineManifest `json:"manifests"`
|
||||
// Skipped lists batch entries whose manifest could not be built (revoked,
|
||||
// deleted from the catalog, access-filtered); the rest of the batch is
|
||||
// still delivered.
|
||||
Skipped []downloads.SkippedManifest `json:"skipped,omitempty"`
|
||||
}
|
||||
|
||||
// downloadCapabilityResponse is the GET /downloads/capability payload clients
|
||||
// use for feature detection instead of introspecting admin settings.
|
||||
type downloadCapabilityResponse struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
DownloadAllowed bool `json:"download_allowed"`
|
||||
QualityPresets []string `json:"quality_presets"`
|
||||
TranscodeEnabled bool `json:"transcode_enabled"`
|
||||
TranscodeUserAllowed bool `json:"transcode_user_allowed"`
|
||||
SeasonDownload bool `json:"season_download"`
|
||||
SeriesMonitoring bool `json:"series_monitoring"`
|
||||
MonitoringModes []string `json:"monitoring_modes,omitempty"`
|
||||
}
|
||||
|
||||
func toDownloadResponse(d *downloads.Download) downloadResponse {
|
||||
resp := downloadResponse{
|
||||
ID: d.ID,
|
||||
ContentID: d.ContentID,
|
||||
EpisodeID: d.EpisodeID,
|
||||
BatchID: d.BatchID,
|
||||
MediaFileID: d.MediaFileID,
|
||||
FileSize: d.FileSize,
|
||||
BytesSent: d.BytesSent,
|
||||
Kind: d.Kind,
|
||||
Status: d.Status,
|
||||
CreatedAt: d.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
ID: d.ID,
|
||||
ContentID: d.ContentID,
|
||||
EpisodeID: d.EpisodeID,
|
||||
BatchID: d.BatchID,
|
||||
DeviceID: d.DeviceID,
|
||||
MediaFileID: d.MediaFileID,
|
||||
FileSize: d.FileSize,
|
||||
BytesSent: d.BytesSent,
|
||||
Kind: d.Kind,
|
||||
Status: d.Status,
|
||||
Quality: d.Quality,
|
||||
EffectiveQuality: d.EffectiveQuality,
|
||||
DeliveryFormat: d.Format,
|
||||
TargetBitrateKbps: d.TargetBitrateKbps,
|
||||
Revision: d.Revision,
|
||||
CreatedAt: d.CreatedAt.Format("2006-01-02T15:04:05Z07:00"),
|
||||
}
|
||||
if d.CompletedAt != nil {
|
||||
s := d.CompletedAt.Format("2006-01-02T15:04:05Z07:00")
|
||||
@@ -83,14 +151,53 @@ func toDownloadResponse(d *download.Download) downloadResponse {
|
||||
return resp
|
||||
}
|
||||
|
||||
// HandleCreateDownload handles POST /downloads.
|
||||
// managedIdentity returns the (profileID, deviceID) the request is acting as.
|
||||
// deviceID comes ONLY from the X-Silo-Device-Id header (never the body/query);
|
||||
// profileID is resolved by the viewer-access middleware from X-Profile-Id.
|
||||
func managedIdentity(r *http.Request) (profileID, deviceID, deviceName, devicePlatform string) {
|
||||
device := deviceMetadataFromRequest(r)
|
||||
return apimw.GetProfileID(r.Context()), device.DeviceID, device.DeviceName, device.DevicePlatform
|
||||
}
|
||||
|
||||
// HandleCapability handles GET /downloads/capability.
|
||||
func (h *DownloadHandler) HandleCapability(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
capInfo, err := h.svc.Capability(r.Context(), userID)
|
||||
if err != nil {
|
||||
slog.Error("failed to load download capability", "user_id", userID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load download capability")
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, downloadCapabilityResponse{
|
||||
Enabled: capInfo.Enabled,
|
||||
DownloadAllowed: capInfo.DownloadAllowed,
|
||||
QualityPresets: capInfo.QualityPresets,
|
||||
TranscodeEnabled: capInfo.TranscodeEnabled,
|
||||
TranscodeUserAllowed: capInfo.TranscodeUserAllowed,
|
||||
SeasonDownload: capInfo.SeasonDownload,
|
||||
SeriesMonitoring: capInfo.SeriesMonitoring,
|
||||
MonitoringModes: capInfo.MonitoringModes,
|
||||
})
|
||||
}
|
||||
|
||||
// HandleCreateDownload handles POST /downloads. The X-Silo-Device-Id header
|
||||
// (if present) makes this a managed device entry; otherwise it is ephemeral.
|
||||
func (h *DownloadHandler) HandleCreateDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
@@ -101,65 +208,102 @@ func (h *DownloadHandler) HandleCreateDownload(w http.ResponseWriter, r *http.Re
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if req.ContentID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "content_id is required")
|
||||
return
|
||||
}
|
||||
|
||||
profileID, deviceID, deviceName, devicePlatform := managedIdentity(r)
|
||||
filter := requestAccessFilter(r)
|
||||
createReq := downloads.CreateRequest{
|
||||
ContentID: req.ContentID,
|
||||
EpisodeID: req.EpisodeID,
|
||||
FileID: req.FileID,
|
||||
Quality: req.Quality,
|
||||
ProfileID: profileID,
|
||||
DeviceID: deviceID,
|
||||
DeviceName: deviceName,
|
||||
DevicePlatform: devicePlatform,
|
||||
}
|
||||
if req.Caps != nil {
|
||||
createReq.Caps = playback.ClientCapabilities{
|
||||
CodecsVideo: req.Caps.CodecsVideo,
|
||||
CodecsAudio: req.Caps.CodecsAudio,
|
||||
AudioPassthroughCodecs: req.Caps.AudioPassthroughCodecs,
|
||||
Containers: req.Caps.Containers,
|
||||
MaxResolution: req.Caps.MaxResolution,
|
||||
HDR: req.Caps.HDR,
|
||||
}
|
||||
}
|
||||
|
||||
if req.Series {
|
||||
downloads, batchID, err := h.svc.CreateQueuedBatch(r.Context(), userID, req.ContentID, filter)
|
||||
var (
|
||||
list []*downloads.Download
|
||||
batchID string
|
||||
skipped []downloads.SkippedDownload
|
||||
err error
|
||||
)
|
||||
// Dispatch on presence, not value: season 0 is the Specials season.
|
||||
if req.Season != nil {
|
||||
if *req.Season < 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "season_number must be >= 0")
|
||||
return
|
||||
}
|
||||
list, batchID, skipped, err = h.svc.CreateSeason(r.Context(), userID, createReq, *req.Season, filter)
|
||||
} else {
|
||||
list, batchID, skipped, err = h.svc.CreateSeries(r.Context(), userID, createReq, filter)
|
||||
}
|
||||
if err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
responses := make([]downloadResponse, 0, len(downloads))
|
||||
for _, d := range downloads {
|
||||
responses := make([]downloadResponse, 0, len(list))
|
||||
for _, d := range list {
|
||||
resp := toDownloadResponse(d)
|
||||
resp.BatchID = batchID
|
||||
if resp.BatchID == "" {
|
||||
resp.BatchID = batchID
|
||||
}
|
||||
responses = append(responses, resp)
|
||||
}
|
||||
writeJSON(w, http.StatusAccepted, downloadsListResponse{Downloads: responses})
|
||||
writeJSON(w, http.StatusAccepted, downloadsListResponse{Downloads: responses, Skipped: skipped})
|
||||
return
|
||||
}
|
||||
|
||||
dl, err := h.svc.CreateQueued(r.Context(), userID, download.CreateRequest{
|
||||
ContentID: req.ContentID,
|
||||
EpisodeID: req.EpisodeID,
|
||||
FileID: req.FileID,
|
||||
}, filter)
|
||||
dl, err := h.svc.Create(r.Context(), userID, createReq, filter)
|
||||
if err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusAccepted, toDownloadResponse(dl))
|
||||
}
|
||||
|
||||
// HandleListDownloads handles GET /downloads.
|
||||
// HandleListDownloads handles GET /downloads. With a device header it returns
|
||||
// the calling device's managed entries; otherwise the user's ephemeral rows.
|
||||
func (h *DownloadHandler) HandleListDownloads(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
downloads, err := h.svc.List(r.Context(), userID)
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
list, err := h.svc.List(r.Context(), userID, profileID, deviceID)
|
||||
if err != nil {
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to list downloads", "user_id", userID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list downloads")
|
||||
return
|
||||
}
|
||||
|
||||
responses := make([]downloadResponse, 0, len(downloads))
|
||||
for _, d := range downloads {
|
||||
responses := make([]downloadResponse, 0, len(list))
|
||||
for _, d := range list {
|
||||
responses = append(responses, toDownloadResponse(d))
|
||||
}
|
||||
writeJSON(w, http.StatusOK, downloadsListResponse{Downloads: responses})
|
||||
@@ -172,28 +316,66 @@ func (h *DownloadHandler) HandleDeleteDownload(w http.ResponseWriter, r *http.Re
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
if err := h.svc.Delete(r.Context(), userID, profileID, deviceID, id); err != nil {
|
||||
if errors.Is(err, downloads.ErrNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to delete download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to delete download")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// HandlePatchDownload handles PATCH /downloads/{id} — a managed-only endpoint
|
||||
// where a client confirms local state (downloading|completed).
|
||||
func (h *DownloadHandler) HandlePatchDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
var req patchDownloadRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.svc.Cancel(r.Context(), userID, id); err != nil {
|
||||
if errors.Is(err, download.ErrNotFound) {
|
||||
if err := h.svc.PatchStatus(r.Context(), userID, profileID, deviceID, id, req.Status); err != nil {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
case errors.Is(err, downloads.ErrInvalidStatus):
|
||||
writeError(w, http.StatusBadRequest, "invalid_status", "status must be downloading or completed")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
default:
|
||||
slog.Error("failed to patch download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to update download")
|
||||
}
|
||||
slog.Error("failed to cancel download", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to cancel download")
|
||||
return
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
@@ -204,41 +386,49 @@ func (h *DownloadHandler) HandleDownloadFile(w http.ResponseWriter, r *http.Requ
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
id := chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.svc.ServeFile(r.Context(), w, r, userID, id); err != nil {
|
||||
if errors.Is(err, download.ErrNotFound) {
|
||||
profileID, deviceID, _, _ := managedIdentity(r)
|
||||
filter := requestAccessFilter(r)
|
||||
if err := h.svc.ServeFile(r.Context(), w, r, userID, profileID, deviceID, id, filter); err != nil {
|
||||
if errors.Is(err, downloads.ErrNotFound) || errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Download not found")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, download.ErrDownloadNotActive) {
|
||||
if errors.Is(err, downloads.ErrDownloadNotActive) {
|
||||
writeError(w, http.StatusConflict, "download_inactive", "This download is no longer active")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrProfileRequired) {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
if errors.Is(err, downloads.ErrFeatureDisabled) || errors.Is(err, downloads.ErrDownloadNotAllowed) {
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
return
|
||||
}
|
||||
slog.Error("failed to serve download file", "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to serve download")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// HandleDirectDownload handles GET /direct-download?file_id=N.
|
||||
// HandleDirectDownload handles GET /direct-download?file_id=N. A legacy
|
||||
// format=original query is accepted, but direct downloads remain original-only.
|
||||
func (h *DownloadHandler) HandleDirectDownload(w http.ResponseWriter, r *http.Request) {
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
@@ -252,23 +442,163 @@ func (h *DownloadHandler) HandleDirectDownload(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
|
||||
filter := requestAccessFilter(r)
|
||||
|
||||
if err := h.svc.ServeDirect(r.Context(), w, r, userID, fileID, filter); err != nil {
|
||||
if err := h.svc.ServeDirect(r.Context(), w, r, userID, fileID, r.URL.Query().Get("format"), filter); err != nil {
|
||||
h.writeDownloadError(w, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// requireManaged validates a managed (device-scoped) request: authentication, a
|
||||
// configured service, and the device + profile identity (device_id from the
|
||||
// X-Silo-Device-Id header only, never the body). On failure it writes the error
|
||||
// response and returns ok=false. Shared by every managed-only endpoint — the
|
||||
// offline assets and the series-monitoring subscriptions.
|
||||
func (h *DownloadHandler) requireManaged(w http.ResponseWriter, r *http.Request) (userID int, profileID, deviceID, deviceName, devicePlatform string, ok bool) {
|
||||
userID = apimw.GetUserID(r.Context())
|
||||
if userID == 0 {
|
||||
writeError(w, http.StatusUnauthorized, "unauthorized", "Authentication required")
|
||||
return
|
||||
}
|
||||
if h.svc == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Downloads not configured")
|
||||
return
|
||||
}
|
||||
profileID, deviceID, deviceName, devicePlatform = managedIdentity(r)
|
||||
if deviceID == "" {
|
||||
writeError(w, http.StatusBadRequest, "device_id_required", "X-Silo-Device-Id header is required")
|
||||
return
|
||||
}
|
||||
if profileID == "" {
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
return
|
||||
}
|
||||
return userID, profileID, deviceID, deviceName, devicePlatform, true
|
||||
}
|
||||
|
||||
// managedAssetIdentity is requireManaged plus the URL {id} param, for the offline
|
||||
// asset endpoints (manifest/artwork/subtitle).
|
||||
func (h *DownloadHandler) managedAssetIdentity(w http.ResponseWriter, r *http.Request) (userID int, profileID, deviceID, id string, ok bool) {
|
||||
userID, profileID, deviceID, _, _, ok = h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return userID, profileID, deviceID, "", false
|
||||
}
|
||||
id = chi.URLParam(r, "id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Download ID is required")
|
||||
return userID, profileID, deviceID, "", false
|
||||
}
|
||||
return userID, profileID, deviceID, id, true
|
||||
}
|
||||
|
||||
// HandleManifest handles GET /downloads/{id}/manifest (managed-only).
|
||||
func (h *DownloadHandler) HandleManifest(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
manifest, err := h.svc.BuildManifest(r.Context(), userID, profileID, deviceID, id, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeAssetError(w, "manifest", id, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, manifest)
|
||||
}
|
||||
|
||||
// HandleBatchManifests handles GET /downloads/batches/{batch_id}/manifests
|
||||
// (managed-only).
|
||||
func (h *DownloadHandler) HandleBatchManifests(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, _, _, ok := h.requireManaged(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
batchID := chi.URLParam(r, "batch_id")
|
||||
if batchID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Batch ID is required")
|
||||
return
|
||||
}
|
||||
manifests, skipped, err := h.svc.BuildBatchManifests(r.Context(), userID, profileID, deviceID, batchID, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
h.writeAssetError(w, "batch_manifests", batchID, err)
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, batchManifestsResponse{Manifests: manifests, Skipped: skipped})
|
||||
}
|
||||
|
||||
// HandleArtwork handles GET /downloads/{id}/artwork/{kind} (managed-only).
|
||||
func (h *DownloadHandler) HandleArtwork(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
kind := chi.URLParam(r, "kind")
|
||||
if err := h.svc.ServeArtwork(r.Context(), w, r, userID, profileID, deviceID, id, kind, requestAccessFilter(r)); err != nil {
|
||||
h.writeAssetError(w, "artwork", id, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// HandleSubtitle handles GET /downloads/{id}/subtitles/{ref} (managed-only).
|
||||
func (h *DownloadHandler) HandleSubtitle(w http.ResponseWriter, r *http.Request) {
|
||||
userID, profileID, deviceID, id, ok := h.managedAssetIdentity(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
ref := chi.URLParam(r, "ref")
|
||||
if err := h.svc.ServeSubtitle(r.Context(), w, r, userID, profileID, deviceID, id, ref, requestAccessFilter(r)); err != nil {
|
||||
h.writeAssetError(w, "subtitle", id, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// writeAssetError maps offline asset (manifest/artwork/subtitle) errors. Access
|
||||
// denials and missing assets collapse to 404 so a download id never reveals the
|
||||
// existence of out-of-scope content.
|
||||
func (h *DownloadHandler) writeAssetError(w http.ResponseWriter, asset, id string, err error) {
|
||||
switch {
|
||||
case errors.Is(err, downloads.ErrNotFound),
|
||||
errors.Is(err, downloads.ErrAssetNotFound),
|
||||
errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Not found")
|
||||
case errors.Is(err, downloads.ErrInvalidSubtitleRef):
|
||||
writeError(w, http.StatusBadRequest, "invalid_subtitle_ref", "Invalid subtitle reference")
|
||||
case errors.Is(err, downloads.ErrDownloadNotActive):
|
||||
writeError(w, http.StatusConflict, "download_inactive", "This download is no longer active")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
case errors.Is(err, downloads.ErrManifestUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Offline assets are not configured")
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled), errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
default:
|
||||
slog.Error("failed to serve download asset", "asset", asset, "download_id", id, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to serve download asset")
|
||||
}
|
||||
}
|
||||
|
||||
func (h *DownloadHandler) writeDownloadError(w http.ResponseWriter, err error) {
|
||||
switch {
|
||||
case errors.Is(err, download.ErrFeatureDisabled):
|
||||
case errors.Is(err, downloads.ErrFeatureDisabled):
|
||||
writeError(w, http.StatusForbidden, "feature_disabled", "Downloads are disabled")
|
||||
case errors.Is(err, download.ErrDownloadNotAllowed):
|
||||
case errors.Is(err, downloads.ErrDownloadNotAllowed):
|
||||
writeError(w, http.StatusForbidden, "forbidden", "You are not allowed to download")
|
||||
case errors.Is(err, download.ErrConcurrentLimitReached):
|
||||
case errors.Is(err, downloads.ErrTranscodeDisabled):
|
||||
writeError(w, http.StatusForbidden, "transcode_disabled", "Download transcoding is disabled")
|
||||
case errors.Is(err, downloads.ErrInvalidQuality):
|
||||
writeError(w, http.StatusBadRequest, "invalid_quality", "Unknown download quality")
|
||||
case errors.Is(err, downloads.ErrProfileRequired):
|
||||
writeError(w, http.StatusBadRequest, "profile_required", "A profile is required for managed downloads")
|
||||
case errors.Is(err, downloads.ErrBulkQualityUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "bulk_quality_unavailable", "Bitrate quality is not available for bulk downloads yet")
|
||||
case errors.Is(err, downloads.ErrQualityUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "quality_unavailable", "This download quality is not available")
|
||||
case errors.Is(err, downloads.ErrFormatUnavailable):
|
||||
writeError(w, http.StatusNotImplemented, "format_unavailable", "This download format is not available yet")
|
||||
case errors.Is(err, downloads.ErrConcurrentLimitReached):
|
||||
writeError(w, http.StatusTooManyRequests, "download_limit_exceeded", "Maximum concurrent downloads reached")
|
||||
case errors.Is(err, download.ErrPeriodLimitReached):
|
||||
case errors.Is(err, downloads.ErrPeriodLimitReached):
|
||||
writeError(w, http.StatusTooManyRequests, "download_quota_exceeded", "Download quota exceeded for this period")
|
||||
case errors.Is(err, downloads.ErrNoDownloadableEpisodes):
|
||||
writeError(w, http.StatusNotFound, "no_downloadable_episodes", "No downloadable episodes found")
|
||||
case errors.Is(err, catalog.ErrItemNotFound):
|
||||
writeError(w, http.StatusNotFound, "not_found", "Media item not found")
|
||||
default:
|
||||
|
||||
@@ -0,0 +1,826 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/auth"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
)
|
||||
|
||||
// fakeDownloadService is a programmable DownloadService for handler tests. It
|
||||
// records the identity (user/profile/device) it was called with so tests can
|
||||
// assert the handler threads header-only device authority correctly.
|
||||
type fakeDownloadService struct {
|
||||
capability downloads.Capability
|
||||
created *downloads.Download
|
||||
createErr error
|
||||
series []*downloads.Download
|
||||
seriesID string
|
||||
seriesErr error
|
||||
list []*downloads.Download
|
||||
listErr error
|
||||
deleteErr error
|
||||
patchErr error
|
||||
serveErr error
|
||||
directErr error
|
||||
manifest *downloads.OfflineManifest
|
||||
batchManifests []*downloads.OfflineManifest
|
||||
batchSkipped []downloads.SkippedManifest
|
||||
manifestErr error
|
||||
artworkErr error
|
||||
subtitleErr error
|
||||
|
||||
gotCreateReq downloads.CreateRequest
|
||||
gotSeriesReq downloads.CreateRequest
|
||||
gotList identityCall
|
||||
gotServe identityCall
|
||||
gotDelete identityCall
|
||||
gotPatch identityCall
|
||||
gotPatchStatus string
|
||||
gotManifest identityCall
|
||||
gotBatchManifest identityCall
|
||||
gotArtwork identityCall
|
||||
gotArtworkKind string
|
||||
gotSubtitle identityCall
|
||||
gotSubtitleRef string
|
||||
gotDirectFormat string
|
||||
gotDirectFileID int
|
||||
|
||||
// Season download + series-monitoring (subscription) fakes.
|
||||
season []*downloads.Download
|
||||
seasonID string
|
||||
seasonErr error
|
||||
gotSeasonReq downloads.CreateRequest
|
||||
gotSeasonNum int
|
||||
seasonCalled bool
|
||||
|
||||
subResult *downloads.SubscriptionResult
|
||||
subList []*downloads.Subscription
|
||||
sub *downloads.Subscription
|
||||
subErr error
|
||||
subDeleteErr error
|
||||
gotSubReq downloads.SubscriptionRequest
|
||||
gotSubPatch downloads.SubscriptionPatch
|
||||
gotSubIdent identityCall
|
||||
syncRegistered int
|
||||
}
|
||||
|
||||
type identityCall struct {
|
||||
userID int
|
||||
profileID string
|
||||
deviceID string
|
||||
downloadID string
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Capability(context.Context, int) (downloads.Capability, error) {
|
||||
return f.capability, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Create(_ context.Context, _ int, req downloads.CreateRequest, _ catalog.AccessFilter) (*downloads.Download, error) {
|
||||
f.gotCreateReq = req
|
||||
if f.createErr != nil {
|
||||
return nil, f.createErr
|
||||
}
|
||||
return f.created, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSeries(_ context.Context, _ int, req downloads.CreateRequest, _ catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error) {
|
||||
f.gotSeriesReq = req
|
||||
if f.seriesErr != nil {
|
||||
return nil, "", nil, f.seriesErr
|
||||
}
|
||||
return f.series, f.seriesID, nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSeason(_ context.Context, _ int, req downloads.CreateRequest, seasonNumber int, _ catalog.AccessFilter) ([]*downloads.Download, string, []downloads.SkippedDownload, error) {
|
||||
f.gotSeasonReq = req
|
||||
f.gotSeasonNum = seasonNumber
|
||||
f.seasonCalled = true
|
||||
if f.seasonErr != nil {
|
||||
return nil, "", nil, f.seasonErr
|
||||
}
|
||||
return f.season, f.seasonID, nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) CreateSubscription(_ context.Context, _ int, req downloads.SubscriptionRequest, _ catalog.AccessFilter) (*downloads.SubscriptionResult, error) {
|
||||
f.gotSubReq = req
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.subResult, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ListSubscriptions(_ context.Context, userID int, profileID, deviceID string) ([]*downloads.Subscription, error) {
|
||||
f.gotSubIdent = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
return f.subList, f.subErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) GetSubscription(_ context.Context, userID int, profileID, deviceID, id string) (*downloads.Subscription, error) {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.sub, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) UpdateSubscription(_ context.Context, userID int, profileID, deviceID, id string, patch downloads.SubscriptionPatch, _ catalog.AccessFilter) (*downloads.SubscriptionResult, error) {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
f.gotSubPatch = patch
|
||||
if f.subErr != nil {
|
||||
return nil, f.subErr
|
||||
}
|
||||
return f.subResult, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) DeleteSubscription(_ context.Context, userID int, profileID, deviceID, id string) error {
|
||||
f.gotSubIdent = identityCall{userID, profileID, deviceID, id}
|
||||
return f.subDeleteErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) SyncSubscriptions(_ context.Context, userID int, profileID, deviceID string, _ catalog.AccessFilter) (int, error) {
|
||||
f.gotSubIdent = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
if f.subErr != nil {
|
||||
return 0, f.subErr
|
||||
}
|
||||
return f.syncRegistered, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeDirect(_ context.Context, w http.ResponseWriter, _ *http.Request, _, fileID int, format string, _ catalog.AccessFilter) error {
|
||||
f.gotDirectFileID = fileID
|
||||
f.gotDirectFormat = format
|
||||
if f.directErr != nil {
|
||||
return f.directErr
|
||||
}
|
||||
_, _ = w.Write([]byte("served"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeFile(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID string, _ catalog.AccessFilter) error {
|
||||
f.gotServe = identityCall{userID, profileID, deviceID, downloadID}
|
||||
if f.serveErr != nil {
|
||||
return f.serveErr
|
||||
}
|
||||
_, _ = w.Write([]byte("served"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) List(_ context.Context, userID int, profileID, deviceID string) ([]*downloads.Download, error) {
|
||||
f.gotList = identityCall{userID: userID, profileID: profileID, deviceID: deviceID}
|
||||
return f.list, f.listErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) Delete(_ context.Context, userID int, profileID, deviceID, downloadID string) error {
|
||||
f.gotDelete = identityCall{userID, profileID, deviceID, downloadID}
|
||||
return f.deleteErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) PatchStatus(_ context.Context, userID int, profileID, deviceID, downloadID, status string) error {
|
||||
f.gotPatch = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotPatchStatus = status
|
||||
return f.patchErr
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) BuildManifest(_ context.Context, userID int, profileID, deviceID, downloadID string, _ catalog.AccessFilter) (*downloads.OfflineManifest, error) {
|
||||
f.gotManifest = identityCall{userID, profileID, deviceID, downloadID}
|
||||
if f.manifestErr != nil {
|
||||
return nil, f.manifestErr
|
||||
}
|
||||
return f.manifest, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) BuildBatchManifests(_ context.Context, userID int, profileID, deviceID, batchID string, _ catalog.AccessFilter) ([]*downloads.OfflineManifest, []downloads.SkippedManifest, error) {
|
||||
f.gotBatchManifest = identityCall{userID, profileID, deviceID, batchID}
|
||||
if f.manifestErr != nil {
|
||||
return nil, nil, f.manifestErr
|
||||
}
|
||||
return f.batchManifests, f.batchSkipped, nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeArtwork(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID, kind string, _ catalog.AccessFilter) error {
|
||||
f.gotArtwork = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotArtworkKind = kind
|
||||
if f.artworkErr != nil {
|
||||
return f.artworkErr
|
||||
}
|
||||
_, _ = w.Write([]byte("img"))
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeDownloadService) ServeSubtitle(_ context.Context, w http.ResponseWriter, _ *http.Request, userID int, profileID, deviceID, downloadID, ref string, _ catalog.AccessFilter) error {
|
||||
f.gotSubtitle = identityCall{userID, profileID, deviceID, downloadID}
|
||||
f.gotSubtitleRef = ref
|
||||
if f.subtitleErr != nil {
|
||||
return f.subtitleErr
|
||||
}
|
||||
_, _ = w.Write([]byte("WEBVTT"))
|
||||
return nil
|
||||
}
|
||||
|
||||
// downloadTestRequest builds a request with auth claims and optional profile +
|
||||
// device identity (as the viewer-access middleware / client headers would set).
|
||||
func downloadTestRequest(method, target string, body []byte, userID int, profileID, deviceID string) *http.Request {
|
||||
var r *http.Request
|
||||
if body != nil {
|
||||
r = httptest.NewRequest(method, target, bytes.NewReader(body))
|
||||
} else {
|
||||
r = httptest.NewRequest(method, target, nil)
|
||||
}
|
||||
if userID != 0 {
|
||||
ctx := apimw.SetClaims(r.Context(), &auth.Claims{
|
||||
UserID: userID,
|
||||
Role: "user",
|
||||
TokenType: auth.TokenTypeAccess,
|
||||
})
|
||||
if profileID != "" {
|
||||
ctx = apimw.SetProfileID(ctx, profileID)
|
||||
}
|
||||
r = r.WithContext(ctx)
|
||||
}
|
||||
if deviceID != "" {
|
||||
r.Header.Set("X-Silo-Device-Id", deviceID)
|
||||
}
|
||||
return r
|
||||
}
|
||||
|
||||
func withChiID(r *http.Request, id string) *http.Request {
|
||||
rctx := chi.NewRouteContext()
|
||||
rctx.URLParams.Add("id", id)
|
||||
return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx))
|
||||
}
|
||||
|
||||
func TestHandleCapability(t *testing.T) {
|
||||
svc := &fakeDownloadService{capability: downloads.Capability{
|
||||
Enabled: true,
|
||||
DownloadAllowed: true,
|
||||
QualityPresets: []string{downloads.QualityOriginal},
|
||||
TranscodeEnabled: false,
|
||||
TranscodeUserAllowed: true,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, downloadTestRequest(http.MethodGet, "/downloads/capability", nil, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp downloadCapabilityResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if !resp.Enabled || !resp.DownloadAllowed || resp.TranscodeEnabled || !resp.TranscodeUserAllowed {
|
||||
t.Fatalf("unexpected capability flags: %+v", resp)
|
||||
}
|
||||
if len(resp.QualityPresets) != 1 || resp.QualityPresets[0] != downloads.QualityOriginal {
|
||||
t.Fatalf("quality presets = %v, want [original]", resp.QualityPresets)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCapabilityUnauthorized(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, httptest.NewRequest(http.MethodGet, "/downloads/capability", nil))
|
||||
if rec.Code != http.StatusUnauthorized {
|
||||
t.Fatalf("status = %d, want 401", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCapabilityNilService(t *testing.T) {
|
||||
h := NewDownloadHandler(nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCapability(rec, downloadTestRequest(http.MethodGet, "/downloads/capability", nil, 7, "", ""))
|
||||
if rec.Code != http.StatusServiceUnavailable {
|
||||
t.Fatalf("status = %d, want 503", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadThreadsQuality(t *testing.T) {
|
||||
svc := &fakeDownloadService{created: &downloads.Download{
|
||||
ID: "dl1", ContentID: "c1", Status: downloads.StatusQueued, Format: downloads.FormatTranscode,
|
||||
Quality: downloads.Quality5Mbps, EffectiveQuality: downloads.Quality5Mbps, TargetBitrateKbps: 5000, Revision: 2,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "c1", Quality: downloads.Quality5Mbps})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotCreateReq.Quality != downloads.Quality5Mbps {
|
||||
t.Fatalf("service received quality %q, want 5mbps", svc.gotCreateReq.Quality)
|
||||
}
|
||||
var resp downloadResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Quality != downloads.Quality5Mbps || resp.DeliveryFormat != downloads.FormatTranscode ||
|
||||
resp.TargetBitrateKbps != 5000 || resp.Revision != 2 {
|
||||
t.Fatalf("response = %+v, want quality/delivery/bitrate/revision", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadSeriesThreadsQuality(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
series: []*downloads.Download{{ID: "dl1", ContentID: "s1", Format: downloads.FormatOriginal, Quality: downloads.QualityOriginal, EffectiveQuality: downloads.QualityOriginal, Revision: 1}},
|
||||
seriesID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Quality: downloads.QualityOriginal})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSeriesReq.Quality != downloads.QualityOriginal {
|
||||
t.Fatalf("series service received quality %q, want original", svc.gotSeriesReq.Quality)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadMissingContentID(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
body, _ := json.Marshal(downloadRequest{Quality: downloads.QualityOriginal})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateDownloadErrorMapping(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantCode int
|
||||
}{
|
||||
{"feature disabled", downloads.ErrFeatureDisabled, http.StatusForbidden},
|
||||
{"not allowed", downloads.ErrDownloadNotAllowed, http.StatusForbidden},
|
||||
{"transcode disabled", downloads.ErrTranscodeDisabled, http.StatusForbidden},
|
||||
{"invalid quality", downloads.ErrInvalidQuality, http.StatusBadRequest},
|
||||
{"quality unavailable", downloads.ErrQualityUnavailable, http.StatusNotImplemented},
|
||||
{"bulk quality unavailable", downloads.ErrBulkQualityUnavailable, http.StatusNotImplemented},
|
||||
{"format unavailable", downloads.ErrFormatUnavailable, http.StatusNotImplemented},
|
||||
{"profile required", downloads.ErrProfileRequired, http.StatusBadRequest},
|
||||
{"concurrent limit", downloads.ErrConcurrentLimitReached, http.StatusTooManyRequests},
|
||||
{"period limit", downloads.ErrPeriodLimitReached, http.StatusTooManyRequests},
|
||||
{"item not found", catalog.ErrItemNotFound, http.StatusNotFound},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{createErr: tc.err})
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "c1"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "", ""))
|
||||
if rec.Code != tc.wantCode {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", rec.Code, tc.wantCode, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestManagedCreateUsesHeaderDeviceNotBody verifies invariant 2's "device_id
|
||||
// authority is the header only": a device_id placed in the JSON body is ignored,
|
||||
// and the service receives the X-Silo-Device-Id header value + the profile.
|
||||
func TestManagedCreateUsesHeaderDeviceNotBody(t *testing.T) {
|
||||
svc := &fakeDownloadService{created: &downloads.Download{ID: "dl1", ContentID: "c1", DeviceID: "devA"}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
// Body smuggles a device_id; it is not a field of downloadRequest, so it is
|
||||
// structurally dropped and must never reach the service.
|
||||
body := []byte(`{"content_id":"c1","device_id":"EVIL","profile_id":"EVIL"}`)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotCreateReq.DeviceID != "devA" {
|
||||
t.Fatalf("service device id = %q, want header value devA", svc.gotCreateReq.DeviceID)
|
||||
}
|
||||
if svc.gotCreateReq.ProfileID != "pA" {
|
||||
t.Fatalf("service profile id = %q, want context value pA", svc.gotCreateReq.ProfileID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedFileThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/file", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDownloadFile(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotServe != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("serve identity = %+v, want {7 pA devA dl1}", svc.gotServe)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedListThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListDownloads(rec, downloadTestRequest(http.MethodGet, "/downloads", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200", rec.Code)
|
||||
}
|
||||
if svc.gotList.deviceID != "devA" || svc.gotList.profileID != "pA" {
|
||||
t.Fatalf("list identity = %+v, want profile pA device devA", svc.gotList)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedDeleteThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodDelete, "/downloads/dl1", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDeleteDownload(rec, req)
|
||||
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotDelete != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("delete identity = %+v, want {7 pA devA dl1}", svc.gotDelete)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDeleteDownloadNotFound(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{deleteErr: downloads.ErrNotFound})
|
||||
rec := httptest.NewRecorder()
|
||||
req := withChiID(downloadTestRequest(http.MethodDelete, "/downloads/dl1", nil, 7, "", ""), "dl1")
|
||||
h.HandleDeleteDownload(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("status = %d, want 404", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedPatchRequiresDeviceHeader(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
// No device header → 400 device_id_required, even with a profile present.
|
||||
req := withChiID(downloadTestRequest(http.MethodPatch, "/downloads/dl1", []byte(`{"status":"completed"}`), 7, "pA", ""), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandlePatchDownload(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 device_id_required (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedPatchThreadsIdentityAndStatus(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodPatch, "/downloads/dl1", []byte(`{"status":"completed"}`), 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandlePatchDownload(rec, req)
|
||||
|
||||
if rec.Code != http.StatusNoContent {
|
||||
t.Fatalf("status = %d, want 204 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotPatch != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("patch identity = %+v, want {7 pA devA dl1}", svc.gotPatch)
|
||||
}
|
||||
if svc.gotPatchStatus != downloads.StatusCompleted {
|
||||
t.Fatalf("patch status = %q, want completed", svc.gotPatchStatus)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDirectDownloadThreadsOriginalFormat(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDirectDownload(rec, downloadTestRequest(http.MethodGet, "/direct-download?file_id=42&format=original", nil, 7, "", ""))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotDirectFileID != 42 {
|
||||
t.Fatalf("file id = %d, want 42", svc.gotDirectFileID)
|
||||
}
|
||||
if svc.gotDirectFormat != downloads.FormatOriginal {
|
||||
t.Fatalf("direct format = %q, want original", svc.gotDirectFormat)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleDirectDownloadMissingFileID(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleDirectDownload(rec, downloadTestRequest(http.MethodGet, "/direct-download", nil, 7, "", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func withChiParams(r *http.Request, params map[string]string) *http.Request {
|
||||
rctx := chi.NewRouteContext()
|
||||
for k, v := range params {
|
||||
rctx.URLParams.Add(k, v)
|
||||
}
|
||||
return r.WithContext(context.WithValue(r.Context(), chi.RouteCtxKey, rctx))
|
||||
}
|
||||
|
||||
func TestManagedManifestRequiresDeviceHeader(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
// Profile present but no device header → 400 device_id_required.
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "pA", ""), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedManifestThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{manifest: &downloads.OfflineManifest{DownloadID: "dl1", Title: "Movie"}}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "pA", "devA"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotManifest != (identityCall{7, "pA", "devA", "dl1"}) {
|
||||
t.Fatalf("manifest identity = %+v, want {7 pA devA dl1}", svc.gotManifest)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedBatchManifestsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{batchManifests: []*downloads.OfflineManifest{
|
||||
{DownloadID: "dl1", Title: "Episode 1"},
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/batches/b1/manifests", nil, 7, "pA", "devA"),
|
||||
map[string]string{"batch_id": "b1"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleBatchManifests(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotBatchManifest != (identityCall{7, "pA", "devA", "b1"}) {
|
||||
t.Fatalf("batch manifest identity = %+v, want {7 pA devA b1}", svc.gotBatchManifest)
|
||||
}
|
||||
var resp batchManifestsResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if len(resp.Manifests) != 1 || resp.Manifests[0].DownloadID != "dl1" {
|
||||
t.Fatalf("manifests = %+v, want one dl1 manifest", resp.Manifests)
|
||||
}
|
||||
}
|
||||
|
||||
// TestManagedAssetsDenyRestrictedProfile is the Phase 2 acceptance test: a
|
||||
// profile denied content access (the service returns ErrItemNotFound) gets a
|
||||
// 404 from manifest, artwork, and subtitle — a download id never reveals
|
||||
// out-of-scope content.
|
||||
func TestManagedAssetsDenyRestrictedProfile(t *testing.T) {
|
||||
t.Run("manifest", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{manifestErr: catalog.ErrItemNotFound})
|
||||
req := withChiID(downloadTestRequest(http.MethodGet, "/downloads/dl1/manifest", nil, 7, "child", "devC"), "dl1")
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleManifest(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("manifest status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
t.Run("artwork", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{artworkErr: catalog.ErrItemNotFound})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/artwork/poster", nil, 7, "child", "devC"),
|
||||
map[string]string{"id": "dl1", "kind": "poster"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleArtwork(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("artwork status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
t.Run("subtitle", func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subtitleErr: catalog.ErrItemNotFound})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/subtitles/external:0", nil, 7, "child", "devC"),
|
||||
map[string]string{"id": "dl1", "ref": "external:0"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSubtitle(rec, req)
|
||||
if rec.Code != http.StatusNotFound {
|
||||
t.Fatalf("subtitle status = %d, want 404", rec.Code)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestManagedArtworkThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{}
|
||||
h := NewDownloadHandler(svc)
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/artwork/backdrop", nil, 7, "pA", "devA"),
|
||||
map[string]string{"id": "dl1", "kind": "backdrop"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleArtwork(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotArtwork != (identityCall{7, "pA", "devA", "dl1"}) || svc.gotArtworkKind != "backdrop" {
|
||||
t.Fatalf("artwork identity = %+v kind = %q", svc.gotArtwork, svc.gotArtworkKind)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagedSubtitleInvalidRef(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subtitleErr: downloads.ErrInvalidSubtitleRef})
|
||||
req := withChiParams(downloadTestRequest(http.MethodGet, "/downloads/dl1/subtitles/bogus", nil, 7, "pA", "devA"),
|
||||
map[string]string{"id": "dl1", "ref": "bogus"})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSubtitle(rec, req)
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func intPtr(i int) *int { return &i }
|
||||
|
||||
// TestHandleCreateDownloadSeasonRoutes verifies that series=true + a season
|
||||
// number routes to CreateSeason (not CreateSeries) with the season threaded.
|
||||
func TestHandleCreateDownloadSeasonRoutes(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
season: []*downloads.Download{{ID: "dl1", ContentID: "s1", EpisodeID: "e1", Format: downloads.FormatOriginal}},
|
||||
seasonID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(2)})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSeasonNum != 2 {
|
||||
t.Fatalf("season number = %d, want 2", svc.gotSeasonNum)
|
||||
}
|
||||
if svc.gotSeasonReq.ContentID != "s1" {
|
||||
t.Fatalf("season content id = %q, want s1", svc.gotSeasonReq.ContentID)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleCreateDownloadSpecialsSeason pins the Specials boundary: season 0
|
||||
// must route to CreateSeason(0), not silently broaden to a full-series
|
||||
// download, and negative seasons are rejected.
|
||||
func TestHandleCreateDownloadSpecialsSeason(t *testing.T) {
|
||||
svc := &fakeDownloadService{
|
||||
season: []*downloads.Download{{ID: "dl1", ContentID: "s1", EpisodeID: "e1", Format: downloads.FormatOriginal}},
|
||||
seasonID: "batch1",
|
||||
}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(0)})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !svc.seasonCalled || svc.gotSeasonNum != 0 {
|
||||
t.Fatalf("seasonCalled=%v num=%d, want CreateSeason(0)", svc.seasonCalled, svc.gotSeasonNum)
|
||||
}
|
||||
if svc.gotSeriesReq.ContentID != "" {
|
||||
t.Fatal("season 0 must not fall through to CreateSeries")
|
||||
}
|
||||
|
||||
rec = httptest.NewRecorder()
|
||||
body, _ = json.Marshal(downloadRequest{ContentID: "s1", Series: true, Season: intPtr(-1)})
|
||||
h.HandleCreateDownload(rec, downloadTestRequest(http.MethodPost, "/downloads", body, 7, "pA", "devA"))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("negative season status = %d, want 400", rec.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleCreateSubscriptionThreadsRequest(t *testing.T) {
|
||||
svc := &fakeDownloadService{subResult: &downloads.SubscriptionResult{
|
||||
Subscription: &downloads.Subscription{ID: "sub1", SeriesID: "s1", Mode: downloads.SubModeLatestSeason, Active: true},
|
||||
Registered: 3,
|
||||
}}
|
||||
h := NewDownloadHandler(svc)
|
||||
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeLatestSeason, DeleteWatched: true, MaxStorageBytes: 1024})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusAccepted {
|
||||
t.Fatalf("status = %d, want 202 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubReq.SeriesID != "s1" || svc.gotSubReq.Mode != downloads.SubModeLatestSeason {
|
||||
t.Fatalf("subscription request = %+v", svc.gotSubReq)
|
||||
}
|
||||
if svc.gotSubReq.DeviceID != "devA" || svc.gotSubReq.ProfileID != "pA" {
|
||||
t.Fatalf("subscription identity = profile %q device %q, want pA/devA", svc.gotSubReq.ProfileID, svc.gotSubReq.DeviceID)
|
||||
}
|
||||
if !svc.gotSubReq.DeleteWatched || svc.gotSubReq.MaxStorageBytes != 1024 {
|
||||
t.Fatalf("subscription options not threaded: %+v", svc.gotSubReq)
|
||||
}
|
||||
var resp subscriptionResultResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Registered != 3 || resp.Subscription.ID != "sub1" {
|
||||
t.Fatalf("unexpected response: %+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleCreateSubscriptionRequiresDevice enforces that monitoring is
|
||||
// device-scoped: a missing X-Silo-Device-Id header is a 400 even with a profile.
|
||||
func TestHandleCreateSubscriptionRequiresDevice(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeAll})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleListSubscriptionsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{subList: []*downloads.Subscription{{ID: "sub1", SeriesID: "s1", Mode: downloads.SubModeAll, Active: true}}}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListSubscriptions(rec, downloadTestRequest(http.MethodGet, "/downloads/subscriptions", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubIdent.profileID != "pA" || svc.gotSubIdent.deviceID != "devA" {
|
||||
t.Fatalf("list identity = %+v, want profile pA device devA", svc.gotSubIdent)
|
||||
}
|
||||
var resp subscriptionsListResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if len(resp.Subscriptions) != 1 || resp.Subscriptions[0].ID != "sub1" {
|
||||
t.Fatalf("unexpected subscriptions: %+v", resp.Subscriptions)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSubscriptionErrorMapping(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
err error
|
||||
wantCode int
|
||||
}{
|
||||
{"not found", downloads.ErrSubscriptionNotFound, http.StatusNotFound},
|
||||
{"unavailable", downloads.ErrSubscriptionsUnavailable, http.StatusServiceUnavailable},
|
||||
{"invalid mode", downloads.ErrInvalidSubscriptionMode, http.StatusBadRequest},
|
||||
{"seasons required", downloads.ErrSeasonsRequired, http.StatusBadRequest},
|
||||
{"not series", downloads.ErrNotSeries, http.StatusBadRequest},
|
||||
{"not allowed", downloads.ErrDownloadNotAllowed, http.StatusForbidden},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{subErr: tc.err})
|
||||
body, _ := json.Marshal(subscriptionRequest{SeriesID: "s1", Mode: downloads.SubModeAll})
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleCreateSubscription(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions", body, 7, "pA", "devA"))
|
||||
if rec.Code != tc.wantCode {
|
||||
t.Fatalf("status = %d, want %d (body: %s)", rec.Code, tc.wantCode, rec.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSyncSubscriptionsThreadsIdentity(t *testing.T) {
|
||||
svc := &fakeDownloadService{syncRegistered: 5}
|
||||
h := NewDownloadHandler(svc)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleSyncSubscriptions(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions/sync", nil, 7, "pA", "devA"))
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if svc.gotSubIdent.profileID != "pA" || svc.gotSubIdent.deviceID != "devA" {
|
||||
t.Fatalf("sync identity = %+v, want profile pA device devA", svc.gotSubIdent)
|
||||
}
|
||||
var resp subscriptionSyncResponse
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode: %v", err)
|
||||
}
|
||||
if resp.Registered != 5 {
|
||||
t.Fatalf("registered = %d, want 5", resp.Registered)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleSyncSubscriptionsRequiresDevice(t *testing.T) {
|
||||
h := NewDownloadHandler(&fakeDownloadService{})
|
||||
rec := httptest.NewRecorder()
|
||||
// Profile present, no device header → 400 device_id_required.
|
||||
h.HandleSyncSubscriptions(rec, downloadTestRequest(http.MethodPost, "/downloads/subscriptions/sync", nil, 7, "pA", ""))
|
||||
if rec.Code != http.StatusBadRequest {
|
||||
t.Fatalf("status = %d, want 400 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
// ConversionHeader tells the client how a kindle-family read was served.
|
||||
// "converted" => body is EPUB; "failed" => body is the raw original and the
|
||||
// client should fall back to opening it externally. Absent => no conversion
|
||||
// was attempted (feature off or non-kindle format), serve/behave as before.
|
||||
const ConversionHeader = "X-Silo-Ebook-Conversion"
|
||||
|
||||
// EbookConverter produces (and caches) a converted EPUB for a source file.
|
||||
// Implemented by *ebookconvert.Cache; an interface here for testability.
|
||||
type EbookConverter interface {
|
||||
GetOrConvert(ctx context.Context, srcPath string, key ebookconvert.SourceKey) (string, error)
|
||||
// Lookup reports a cached conversion WITHOUT converting (used by HEAD so a
|
||||
// metadata probe never kicks off a full conversion): a path on hit, a
|
||||
// non-nil error for a negatively-cached source, else a miss.
|
||||
Lookup(key ebookconvert.SourceKey) (string, error, bool)
|
||||
}
|
||||
|
||||
// EbookConversion bundles the converter with the admin-flag predicate. A nil
|
||||
// *EbookConversion (or nil Converter) means the feature is off.
|
||||
type EbookConversion struct {
|
||||
Converter EbookConverter
|
||||
// Enabled reports whether the admin flag is on for this request.
|
||||
Enabled func(ctx context.Context) bool
|
||||
}
|
||||
|
||||
func (c *EbookConversion) active(ctx context.Context) bool {
|
||||
return c != nil && c.Converter != nil && c.Enabled != nil && c.Enabled(ctx)
|
||||
}
|
||||
|
||||
// kindleReaderFormats are the Kindle-family formats the converter handles.
|
||||
var kindleReaderFormats = map[string]bool{"mobi": true, "azw": true, "azw3": true}
|
||||
|
||||
func isKindleReaderFile(file *models.MediaFile) bool {
|
||||
return file != nil && kindleReaderFormats[ebookReaderFormat(file.FilePath, file.Container)]
|
||||
}
|
||||
|
||||
// serveEbook serves an ebook file, converting Kindle-family formats to EPUB
|
||||
// when the feature is enabled. On conversion failure it serves the raw original
|
||||
// with the ConversionHeader=failed contract so the client opens it externally.
|
||||
func (h *EbookReaderHandler) serveEbook(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
|
||||
if h.Conversion.active(r.Context()) && isKindleReaderFile(file) {
|
||||
// Strong identity for both the conversion cache and the response ETag:
|
||||
// id + size + mtime (stat) + content hash + module version.
|
||||
key, statErr := ebookconvert.SourceKeyFromStat(file.ID, file.FilePath)
|
||||
if statErr != nil {
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
key.Checksum = file.FileHash
|
||||
|
||||
// HEAD must not trigger a (possibly minute-long, 1 GiB) conversion. Answer
|
||||
// from cache only; on a miss, advertise the converted representation
|
||||
// cheaply and let the GET produce the body + authoritative verdict.
|
||||
if r.Method == http.MethodHead {
|
||||
return h.serveKindleHead(w, r, file, key)
|
||||
}
|
||||
|
||||
epubPath, err := h.Conversion.Converter.GetOrConvert(r.Context(), file.FilePath, key)
|
||||
switch {
|
||||
case err == nil:
|
||||
if serveErr := serveConvertedEpub(w, r, file, epubPath, key); serveErr != nil {
|
||||
// Produced but unservable (open/stat) — honor the contract, not a 500.
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
return nil
|
||||
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
|
||||
// Caller went away / request cancelled — not a conversion verdict.
|
||||
return err
|
||||
default:
|
||||
// DRM, corrupt, oversize, timed out, unavailable: fall back to the raw
|
||||
// original and tell the client so it can open externally.
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
}
|
||||
// Feature off or non-kindle format: serve the raw original with its native
|
||||
// MIME and no conversion header. This path deliberately omits the failed
|
||||
// contract — when conversion is disabled the capability endpoint already
|
||||
// reports `enabled:false`, so clients keep their external-open path and never
|
||||
// expect an EPUB here. (No representation flip to guard against.)
|
||||
return serveEbookInline(w, r, file)
|
||||
}
|
||||
|
||||
// serveRawKindleFallback serves the raw original with the failed-conversion
|
||||
// contract. no-store because the same URL flips raw/converted with the flag.
|
||||
func (h *EbookReaderHandler) serveRawKindleFallback(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
|
||||
w.Header().Set(ConversionHeader, "failed")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
return serveEbookInline(w, r, file)
|
||||
}
|
||||
|
||||
// serveKindleHead answers a HEAD for a kindle-family file without converting:
|
||||
// - cache hit → real converted headers (and ServeContent's Content-Length);
|
||||
// - neg-cached → the failed contract (raw original headers);
|
||||
// - miss → converted headers advertised optimistically (no body, no
|
||||
// Content-Length, no conversion). A HEAD is only a hint; the GET delivers
|
||||
// the body and the authoritative DRM/failure verdict, which then populates
|
||||
// the (positive or negative) cache that subsequent HEADs read.
|
||||
func (h *EbookReaderHandler) serveKindleHead(w http.ResponseWriter, r *http.Request, file *models.MediaFile, key ebookconvert.SourceKey) error {
|
||||
path, lookupErr, ok := h.Conversion.Converter.Lookup(key)
|
||||
switch {
|
||||
case ok:
|
||||
if serveErr := serveConvertedEpub(w, r, file, path, key); serveErr != nil {
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
return nil
|
||||
case lookupErr != nil:
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
default:
|
||||
setConvertedEpubHeaders(w, file, key)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// setConvertedEpubHeaders writes the headers for a converted-EPUB response.
|
||||
// Because the same URL can return raw or converted bytes depending on the
|
||||
// flag/outcome, the ETag is derived from the exact conversion cache key (so a
|
||||
// validator can never disagree with the cache) with a revalidate policy that
|
||||
// keeps clients/proxies from serving a stale representation.
|
||||
func setConvertedEpubHeaders(w http.ResponseWriter, file *models.MediaFile, key ebookconvert.SourceKey) {
|
||||
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
|
||||
w.Header().Set("Content-Type", "application/epub+zip")
|
||||
w.Header().Set("Content-Disposition", inlineContentDisposition(name))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set(ConversionHeader, "converted")
|
||||
w.Header().Set("ETag", fmt.Sprintf("%q", "epubconv-"+key.CacheKey()))
|
||||
w.Header().Set("Cache-Control", "private, max-age=0, must-revalidate")
|
||||
}
|
||||
|
||||
// serveConvertedEpub serves the cached EPUB with EPUB headers (see
|
||||
// setConvertedEpubHeaders). For a HEAD, ServeContent emits the headers and
|
||||
// Content-Length without a body.
|
||||
func serveConvertedEpub(w http.ResponseWriter, r *http.Request, file *models.MediaFile, epubPath string, key ebookconvert.SourceKey) error {
|
||||
f, err := os.Open(epubPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening converted epub: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat converted epub: %w", err)
|
||||
}
|
||||
|
||||
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
|
||||
setConvertedEpubHeaders(w, file, key)
|
||||
http.ServeContent(w, r, name, stat.ModTime(), f)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ebookConversionCapability is the client-facing advertisement for the
|
||||
// Kindle->EPUB feature. Clients flip mobi/azw/azw3 to in-app reading only when
|
||||
// Enabled is true; otherwise they keep the external-open path.
|
||||
type ebookConversionCapability struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
SourceFormats []string `json:"source_formats"`
|
||||
ServedFormat string `json:"served_format"`
|
||||
Header string `json:"header"`
|
||||
HeaderOnFailed string `json:"header_failed_value"`
|
||||
}
|
||||
|
||||
// HandleConversionCapability reports whether Kindle->EPUB conversion is active
|
||||
// (admin flag on AND the converter initialized). GET /api/v1/ebooks/capability.
|
||||
func (h *EbookReaderHandler) HandleConversionCapability(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, ebookConversionCapability{
|
||||
Enabled: h.Conversion.active(r.Context()),
|
||||
SourceFormats: []string{"mobi", "azw", "azw3"},
|
||||
ServedFormat: "epub",
|
||||
Header: ConversionHeader,
|
||||
HeaderOnFailed: "failed",
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeConverter struct {
|
||||
epubPath string
|
||||
err error
|
||||
calls int
|
||||
|
||||
lookupPath string
|
||||
lookupErr error
|
||||
lookupOK bool
|
||||
}
|
||||
|
||||
func (f *fakeConverter) GetOrConvert(_ context.Context, _ string, _ ebookconvert.SourceKey) (string, error) {
|
||||
f.calls++
|
||||
return f.epubPath, f.err
|
||||
}
|
||||
|
||||
func (f *fakeConverter) Lookup(_ ebookconvert.SourceKey) (string, error, bool) {
|
||||
return f.lookupPath, f.lookupErr, f.lookupOK
|
||||
}
|
||||
|
||||
func writeTemp(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func serveAndRecord(t *testing.T, h *EbookReaderHandler, file *models.MediaFile) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook: %v", err)
|
||||
}
|
||||
return rec
|
||||
}
|
||||
|
||||
func enabledConversion(conv EbookConverter, on bool) *EbookConversion {
|
||||
return &EbookConversion{Converter: conv, Enabled: func(context.Context) bool { return on }}
|
||||
}
|
||||
|
||||
func TestServeEbook_ConvertsKindleWhenEnabled(t *testing.T) {
|
||||
epub := writeTemp(t, "out.epub", "PK\x03\x04 fake-epub-bytes")
|
||||
src := writeTemp(t, "book.azw3", "raw-azw3-source")
|
||||
conv := &fakeConverter{epubPath: epub}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3", FileHash: "abc123"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
|
||||
}
|
||||
if rec.Header().Get("ETag") == "" {
|
||||
t.Fatal("expected an ETag on converted response")
|
||||
}
|
||||
if rec.Body.String() != "PK\x03\x04 fake-epub-bytes" {
|
||||
t.Fatalf("body = %q, want the epub bytes", rec.Body.String())
|
||||
}
|
||||
if conv.calls != 1 {
|
||||
t.Fatalf("converter called %d times, want 1", conv.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_ConversionFailureFallsBackToRawWithHeader(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
|
||||
conv := &fakeConverter{err: ebookconvert.ErrDRMProtected}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 2, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get(ConversionHeader); got != "failed" {
|
||||
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
|
||||
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
|
||||
}
|
||||
if rec.Body.String() != "raw-mobi-bytes" {
|
||||
t.Fatalf("body = %q, want raw mobi bytes", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_DisabledServesRaw(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
|
||||
conv := &fakeConverter{epubPath: "should-not-be-used"}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, false)}
|
||||
|
||||
file := &models.MediaFile{ID: 3, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get(ConversionHeader); got != "" {
|
||||
t.Fatalf("%s = %q, want empty (no conversion attempted)", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
|
||||
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("converter called %d times while disabled, want 0", conv.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_NonKindleUntouched(t *testing.T) {
|
||||
raw := writeTemp(t, "book.epub", "epub-bytes")
|
||||
conv := &fakeConverter{epubPath: "should-not-be-used"}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 4, FilePath: raw, Container: "epub"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if rec.Header().Get(ConversionHeader) != "" {
|
||||
t.Fatal("non-kindle file must not trigger conversion")
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("converter called %d times for epub, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want epub", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_NilConversionServesRaw(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw")
|
||||
h := &EbookReaderHandler{} // Conversion nil
|
||||
file := &models.MediaFile{ID: 5, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
if rec.Header().Get(ConversionHeader) != "" {
|
||||
t.Fatal("nil Conversion must not set the conversion header")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleConversionCapability(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
conv *EbookConversion
|
||||
want bool
|
||||
}{
|
||||
{"enabled", enabledConversion(&fakeConverter{}, true), true},
|
||||
{"flag-off", enabledConversion(&fakeConverter{}, false), false},
|
||||
{"not-wired", nil, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := &EbookReaderHandler{Conversion: tc.conv}
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleConversionCapability(rec, httptest.NewRequest(http.MethodGet, "/ebooks/capability", nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d", rec.Code)
|
||||
}
|
||||
body := rec.Body.String()
|
||||
wantField := `"enabled":false`
|
||||
if tc.want {
|
||||
wantField = `"enabled":true`
|
||||
}
|
||||
if !strings.Contains(body, wantField) {
|
||||
t.Fatalf("body %q missing %s", body, wantField)
|
||||
}
|
||||
if !strings.Contains(body, `"served_format":"epub"`) || !strings.Contains(body, "mobi") {
|
||||
t.Fatalf("capability body incomplete: %s", body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD on a cache miss must NOT trigger a (possibly minute-long) conversion;
|
||||
// it advertises the converted representation cheaply and lets the GET do the work.
|
||||
func TestServeEbook_HeadDoesNotConvertOnMiss(t *testing.T) {
|
||||
src := writeTemp(t, "book.azw3", "raw-azw3-source")
|
||||
conv := &fakeConverter{lookupOK: false} // cache miss
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a cache miss triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted (advertised on HEAD miss)", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
if rec.Header().Get("ETag") == "" {
|
||||
t.Fatal("HEAD miss should still carry the conversion ETag")
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD that hits the cache serves the converted headers from the cached file,
|
||||
// without converting.
|
||||
func TestServeEbook_HeadServesCachedConverted(t *testing.T) {
|
||||
epub := writeTemp(t, "out.epub", "PK\x03\x04 cached-epub")
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{lookupPath: epub, lookupOK: true}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 2, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a cache hit triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD on a negatively-cached (DRM/failed) source reports the failed contract
|
||||
// without converting.
|
||||
func TestServeEbook_HeadDRMReturnsFailed(t *testing.T) {
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{lookupErr: ebookconvert.ErrDRMProtected}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 3, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a negatively-cached source triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "failed" {
|
||||
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_ContextCancelPropagates(t *testing.T) {
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{err: context.Canceled}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 6, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, "/read", nil)
|
||||
err := h.serveEbook(rec, req, file)
|
||||
if err == nil {
|
||||
t.Fatal("expected context cancellation to propagate, not fall back")
|
||||
}
|
||||
if rec.Header().Get(ConversionHeader) == "failed" {
|
||||
t.Fatal("cancellation must not be reported as a conversion failure")
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,9 @@ type EbookReaderHandler struct {
|
||||
ProgressStore EbookReaderProgressStore
|
||||
ConfigStore EbookReaderConfigStore
|
||||
AnnotationStore EbookReaderAnnotationStore
|
||||
// Conversion, when set and enabled, transparently converts Kindle-family
|
||||
// files to EPUB on read. Nil means the feature is off.
|
||||
Conversion *EbookConversion
|
||||
}
|
||||
|
||||
func NewEbookReaderHandler(authorizer *MediaFileAuthorizer) *EbookReaderHandler {
|
||||
@@ -130,7 +133,7 @@ func (h *EbookReaderHandler) HandleReadFile(w http.ResponseWriter, r *http.Reque
|
||||
return
|
||||
}
|
||||
|
||||
if err := serveEbookInline(w, r, file); err != nil {
|
||||
if err := h.serveEbook(w, r, file); err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Ebook file not found")
|
||||
return
|
||||
@@ -1046,15 +1049,25 @@ func (s *PGEbookReaderProgressStore) Upsert(ctx context.Context, progress EbookR
|
||||
if s == nil || s.pool == nil {
|
||||
return fmt.Errorf("ebook reader progress store is not configured")
|
||||
}
|
||||
if _, err := s.pool.Exec(ctx, `
|
||||
// A routine autosave (e.g. reopening a finished book) must not silently drop
|
||||
// a "finished" item below the threshold and un-mark it read; once finished,
|
||||
// progress only moves on an explicit unread (which deletes the row). Below
|
||||
// the threshold, progress tracks freely. Manga chapter ✓ marks ride on this
|
||||
// same row, so the guard protects them too.
|
||||
query := fmt.Sprintf(`
|
||||
INSERT INTO ebook_reader_progress
|
||||
(user_id, profile_id, content_id, file_id, location, progress, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (user_id, profile_id, content_id) DO UPDATE SET
|
||||
file_id = EXCLUDED.file_id,
|
||||
location = EXCLUDED.location,
|
||||
progress = EXCLUDED.progress,
|
||||
updated_at = EXCLUDED.updated_at`,
|
||||
progress = CASE
|
||||
WHEN ebook_reader_progress.progress >= %[1]v AND EXCLUDED.progress < %[1]v
|
||||
THEN ebook_reader_progress.progress
|
||||
ELSE EXCLUDED.progress
|
||||
END,
|
||||
updated_at = EXCLUDED.updated_at`, models.EbookFinishedProgressThreshold)
|
||||
if _, err := s.pool.Exec(ctx, query,
|
||||
progress.UserID,
|
||||
progress.ProfileID,
|
||||
progress.ContentID,
|
||||
|
||||
@@ -28,8 +28,10 @@ type taskInfoLister interface {
|
||||
ListTasks(includeHidden bool) []taskmanager.TaskInfo
|
||||
}
|
||||
|
||||
const maxRealtimeScanSnapshotRuns = 500
|
||||
|
||||
type activeScanLister interface {
|
||||
ListActive(ctx context.Context) ([]evt.ScanRun, error)
|
||||
ListActiveSnapshot(ctx context.Context, limit int) ([]evt.ScanRun, error)
|
||||
}
|
||||
|
||||
type EventsHandler struct {
|
||||
@@ -414,16 +416,21 @@ func (h *EventsHandler) snapshotForChannel(
|
||||
if h == nil {
|
||||
return json.RawMessage("[]"), nil
|
||||
}
|
||||
runs := make([]evt.ScanRun, 0)
|
||||
if h.persistedScans != nil {
|
||||
persisted, err := h.persistedScans.ListActive(r.Context())
|
||||
runs := make([]evt.ScanRun, 0, maxRealtimeScanSnapshotRuns)
|
||||
remaining := maxRealtimeScanSnapshotRuns
|
||||
if h.persistedScans != nil && remaining > 0 {
|
||||
persisted, err := h.persistedScans.ListActiveSnapshot(r.Context(), remaining)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if len(persisted) > remaining {
|
||||
persisted = persisted[:remaining]
|
||||
}
|
||||
runs = append(runs, persisted...)
|
||||
remaining -= len(persisted)
|
||||
}
|
||||
if h.scans != nil {
|
||||
runs = append(runs, h.scans.ListActive()...)
|
||||
if h.scans != nil && remaining > 0 {
|
||||
runs = append(runs, h.scans.ListActiveLimit(remaining)...)
|
||||
}
|
||||
return marshalJSON(runs), nil
|
||||
case evt.ChannelHistoryImport:
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
)
|
||||
|
||||
type fakeActiveScanSnapshotLister struct {
|
||||
runs []evt.ScanRun
|
||||
limit int
|
||||
}
|
||||
|
||||
func (f *fakeActiveScanSnapshotLister) ListActiveSnapshot(_ context.Context, limit int) ([]evt.ScanRun, error) {
|
||||
f.limit = limit
|
||||
if limit > 0 && len(f.runs) > limit {
|
||||
return f.runs[:limit], nil
|
||||
}
|
||||
return f.runs, nil
|
||||
}
|
||||
|
||||
func TestScanSnapshotCapsActiveRuns(t *testing.T) {
|
||||
persisted := &fakeActiveScanSnapshotLister{runs: makeScanRuns("persisted-", 300)}
|
||||
registry := evt.NewScanRegistry()
|
||||
for _, run := range makeScanRuns("registry-", 250) {
|
||||
registry.Upsert(run)
|
||||
}
|
||||
registry.Upsert(evt.ScanRun{ID: "registry-completed", Status: "completed"})
|
||||
|
||||
handler := &EventsHandler{
|
||||
persistedScans: persisted,
|
||||
scans: registry,
|
||||
}
|
||||
raw, err := handler.snapshotForChannel(httptest.NewRequest("GET", "/", nil), nil, "", evt.ChannelScans)
|
||||
if err != nil {
|
||||
t.Fatalf("snapshotForChannel: %v", err)
|
||||
}
|
||||
|
||||
var runs []evt.ScanRun
|
||||
if err := json.Unmarshal(raw, &runs); err != nil {
|
||||
t.Fatalf("decode snapshot: %v", err)
|
||||
}
|
||||
if len(runs) != maxRealtimeScanSnapshotRuns {
|
||||
t.Fatalf("snapshot length = %d, want %d", len(runs), maxRealtimeScanSnapshotRuns)
|
||||
}
|
||||
if persisted.limit != maxRealtimeScanSnapshotRuns {
|
||||
t.Fatalf("persisted snapshot limit = %d, want %d", persisted.limit, maxRealtimeScanSnapshotRuns)
|
||||
}
|
||||
|
||||
registryRuns := 0
|
||||
for _, run := range runs {
|
||||
if strings.HasPrefix(run.ID, "registry-") {
|
||||
registryRuns++
|
||||
}
|
||||
}
|
||||
if registryRuns != maxRealtimeScanSnapshotRuns-len(persisted.runs) {
|
||||
t.Fatalf("registry runs in snapshot = %d, want %d", registryRuns, maxRealtimeScanSnapshotRuns-len(persisted.runs))
|
||||
}
|
||||
}
|
||||
|
||||
func makeScanRuns(prefix string, count int) []evt.ScanRun {
|
||||
runs := make([]evt.ScanRun, 0, count)
|
||||
for i := 0; i < count; i++ {
|
||||
runs = append(runs, evt.ScanRun{
|
||||
ID: prefix + strconv.Itoa(i),
|
||||
LibraryID: 1,
|
||||
Mode: "file",
|
||||
Path: "/library/file-" + strconv.Itoa(i) + ".mkv",
|
||||
Trigger: "autoscan",
|
||||
Status: "accepted",
|
||||
})
|
||||
}
|
||||
return runs
|
||||
}
|
||||
@@ -25,8 +25,8 @@ type personalDataItemRepository interface {
|
||||
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
type LocalFavoriteEventDispatcher interface {
|
||||
HandleLocalFavoriteEvent(ctx context.Context, event watchsync.LocalFavoriteEvent) error
|
||||
type LocalListEventDispatcher interface {
|
||||
HandleLocalListEvent(ctx context.Context, event watchsync.LocalListEvent) error
|
||||
}
|
||||
|
||||
// PersonalDataHandler handles favorites, watchlist, and history endpoints.
|
||||
@@ -37,7 +37,7 @@ type PersonalDataHandler struct {
|
||||
seasonRepo *catalog.SeasonRepository
|
||||
detailSvc *catalog.DetailService
|
||||
EventsHub *evt.Hub
|
||||
localFavoriteDispatcher LocalFavoriteEventDispatcher
|
||||
localListDispatcher LocalListEventDispatcher
|
||||
profileStaler ProfileStaler
|
||||
profileRefreshRequester ProfileRefreshRequester
|
||||
ebookProgressStore EbookReaderProgressLister
|
||||
@@ -78,8 +78,8 @@ func (h *PersonalDataHandler) SetSeasonRepo(repo *catalog.SeasonRepository) {
|
||||
h.seasonRepo = repo
|
||||
}
|
||||
|
||||
func (h *PersonalDataHandler) SetLocalFavoriteEventDispatcher(dispatcher LocalFavoriteEventDispatcher) {
|
||||
h.localFavoriteDispatcher = dispatcher
|
||||
func (h *PersonalDataHandler) SetLocalListEventDispatcher(dispatcher LocalListEventDispatcher) {
|
||||
h.localListDispatcher = dispatcher
|
||||
}
|
||||
|
||||
// --- Response types ---
|
||||
@@ -219,7 +219,7 @@ func (h *PersonalDataHandler) HandleAddFavorite(w http.ResponseWriter, r *http.R
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventAdded, userID, profileID, itemID)
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeAdded, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
|
||||
IsFavorite: boolPtr(true),
|
||||
@@ -249,7 +249,7 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalFavoriteEvent(r.Context(), watchsync.LocalFavoriteEventRemoved, userID, profileID, itemID)
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindFavorites, watchsync.ListChangeRemoved, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "favorite", userStateEventState{
|
||||
IsFavorite: boolPtr(false),
|
||||
@@ -257,15 +257,15 @@ func (h *PersonalDataHandler) HandleRemoveFavorite(w http.ResponseWriter, r *htt
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, kind watchsync.LocalFavoriteEventKind, userID int, profileID string, itemID string) {
|
||||
if h == nil || h.localFavoriteDispatcher == nil || h.itemRepo == nil {
|
||||
func (h *PersonalDataHandler) dispatchLocalListEvent(ctx context.Context, list watchsync.ListKind, change watchsync.ListChange, userID int, profileID string, itemID string) {
|
||||
if h == nil || h.localListDispatcher == nil || h.itemRepo == nil {
|
||||
return
|
||||
}
|
||||
item, err := h.itemRepo.GetByID(ctx, itemID)
|
||||
if err != nil || item == nil {
|
||||
return
|
||||
}
|
||||
favorite := watchsync.LocalFavorite{
|
||||
listItem := watchsync.LocalFavorite{
|
||||
MediaItemID: item.ContentID,
|
||||
Kind: item.Type,
|
||||
Title: item.Title,
|
||||
@@ -275,11 +275,12 @@ func (h *PersonalDataHandler) dispatchLocalFavoriteEvent(ctx context.Context, ki
|
||||
TVDBID: item.TvdbID,
|
||||
FavoritedAt: time.Now().UTC(),
|
||||
}
|
||||
_ = h.localFavoriteDispatcher.HandleLocalFavoriteEvent(ctx, watchsync.LocalFavoriteEvent{
|
||||
Kind: kind,
|
||||
_ = h.localListDispatcher.HandleLocalListEvent(ctx, watchsync.LocalListEvent{
|
||||
List: list,
|
||||
Change: change,
|
||||
UserID: userID,
|
||||
ProfileID: profileID,
|
||||
Favorites: []watchsync.LocalFavorite{favorite},
|
||||
Items: []watchsync.LocalFavorite{listItem},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -374,6 +375,7 @@ func (h *PersonalDataHandler) HandleAddToWatchlist(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeAdded, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
|
||||
InWatchlist: boolPtr(true),
|
||||
@@ -403,6 +405,7 @@ func (h *PersonalDataHandler) HandleRemoveFromWatchlist(w http.ResponseWriter, r
|
||||
return
|
||||
}
|
||||
|
||||
h.dispatchLocalListEvent(r.Context(), watchsync.ListKindWatchlist, watchsync.ListChangeRemoved, userID, profileID, itemID)
|
||||
triggerProfileRefresh(r.Context(), h.profileStaler, h.profileRefreshRequester, userID, profileID)
|
||||
publishUserStateEvent(r.Context(), h.EventsHub, userID, profileID, itemID, "", "watchlist", userStateEventState{
|
||||
InWatchlist: boolPtr(false),
|
||||
|
||||
@@ -56,14 +56,27 @@ func (h *FilesystemHandler) HandleBrowse(w http.ResponseWriter, r *http.Request)
|
||||
|
||||
result := make([]filesystemBrowseEntry, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
full := filepath.Join(cleaned, name)
|
||||
|
||||
// os.ReadDir reports lstat semantics, so a symlink to a directory has
|
||||
// IsDir() == false. Follow symlink entries with os.Stat and keep them only
|
||||
// when the target is a directory; skip links whose target cannot be stat'd
|
||||
// (broken or looping). Non-symlink, non-directory entries (regular files)
|
||||
// are skipped without the extra stat so large media folders stay fast.
|
||||
if !entry.IsDir() {
|
||||
continue
|
||||
if entry.Type()&os.ModeSymlink == 0 {
|
||||
continue
|
||||
}
|
||||
target, err := os.Stat(full)
|
||||
if err != nil || !target.IsDir() {
|
||||
continue
|
||||
}
|
||||
}
|
||||
|
||||
name := entry.Name()
|
||||
result = append(result, filesystemBrowseEntry{
|
||||
Name: name,
|
||||
Path: filepath.Join(cleaned, name),
|
||||
Path: full,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// TestHandleBrowseFollowsDirectorySymlinks verifies that the folder browser
|
||||
// lists symlinks that resolve to directories (issue #208). os.ReadDir returns
|
||||
// lstat-based entries, so a directory symlink reports IsDir() == false and was
|
||||
// silently dropped; the handler must os.Stat each entry to follow the link.
|
||||
func TestHandleBrowseFollowsDirectorySymlinks(t *testing.T) {
|
||||
if runtime.GOOS == "windows" {
|
||||
t.Skip("symlink creation is unreliable on Windows CI")
|
||||
}
|
||||
t.Parallel()
|
||||
|
||||
root := t.TempDir()
|
||||
|
||||
// A real directory that should always be listed.
|
||||
realDir := filepath.Join(root, "real-dir")
|
||||
if err := os.Mkdir(realDir, 0o755); err != nil {
|
||||
t.Fatalf("mkdir real-dir: %v", err)
|
||||
}
|
||||
|
||||
// The symlink-to-directory target lives outside root so the browser cannot
|
||||
// reach it except by following the link — mirrors the reported setup where
|
||||
// entries under a bind mount point into a separate FUSE mount.
|
||||
linkTarget := filepath.Join(t.TempDir(), "link-target")
|
||||
if err := os.Mkdir(linkTarget, 0o755); err != nil {
|
||||
t.Fatalf("mkdir link-target: %v", err)
|
||||
}
|
||||
dirSymlink := filepath.Join(root, "symlinked-dir")
|
||||
if err := os.Symlink(linkTarget, dirSymlink); err != nil {
|
||||
t.Fatalf("symlink dir: %v", err)
|
||||
}
|
||||
|
||||
// A regular file: must never be listed by the folder browser.
|
||||
regularFile := filepath.Join(root, "note.txt")
|
||||
if err := os.WriteFile(regularFile, []byte("x"), 0o644); err != nil {
|
||||
t.Fatalf("write file: %v", err)
|
||||
}
|
||||
|
||||
// A symlink to a file: resolves to a non-directory, must not be listed.
|
||||
fileSymlink := filepath.Join(root, "note-link")
|
||||
if err := os.Symlink(regularFile, fileSymlink); err != nil {
|
||||
t.Fatalf("symlink file: %v", err)
|
||||
}
|
||||
|
||||
// A broken symlink: os.Stat fails; it must be skipped, not error the request.
|
||||
brokenSymlink := filepath.Join(root, "broken-link")
|
||||
if err := os.Symlink(filepath.Join(root, "does-not-exist"), brokenSymlink); err != nil {
|
||||
t.Fatalf("symlink broken: %v", err)
|
||||
}
|
||||
|
||||
handler := NewFilesystemHandler()
|
||||
req := httptest.NewRequest(http.MethodGet, "/filesystem/browse?path="+root, nil)
|
||||
rec := httptest.NewRecorder()
|
||||
handler.HandleBrowse(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rec.Code, rec.Body.String())
|
||||
}
|
||||
|
||||
var resp filesystemBrowseResponse
|
||||
if err := json.NewDecoder(rec.Body).Decode(&resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
|
||||
got := make(map[string]string, len(resp.Entries))
|
||||
for _, e := range resp.Entries {
|
||||
got[e.Name] = e.Path
|
||||
}
|
||||
|
||||
if _, ok := got["real-dir"]; !ok {
|
||||
t.Errorf("real-dir missing from entries: %+v", resp.Entries)
|
||||
}
|
||||
if path, ok := got["symlinked-dir"]; !ok {
|
||||
t.Errorf("symlinked-dir (directory symlink) missing from entries: %+v", resp.Entries)
|
||||
} else if want := filepath.Join(root, "symlinked-dir"); path != want {
|
||||
t.Errorf("symlinked-dir path = %q, want %q", path, want)
|
||||
}
|
||||
if _, ok := got["note.txt"]; ok {
|
||||
t.Errorf("regular file note.txt should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
if _, ok := got["note-link"]; ok {
|
||||
t.Errorf("file symlink note-link should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
if _, ok := got["broken-link"]; ok {
|
||||
t.Errorf("broken symlink broken-link should not be listed: %+v", resp.Entries)
|
||||
}
|
||||
}
|
||||
+281
-85
@@ -38,6 +38,11 @@ type batchEpisodeFileProvider interface {
|
||||
ListByEpisodeIDs(ctx context.Context, episodeIDs []string) (map[string][]*models.MediaFile, error)
|
||||
}
|
||||
|
||||
type overlayFileProvider interface {
|
||||
ListOverlayFilesByContentIDs(ctx context.Context, contentIDs []string) (map[string][]*models.MediaFile, error)
|
||||
ListOverlayFilesByEpisodeIDs(ctx context.Context, episodeIDs []string) (map[string][]*models.MediaFile, error)
|
||||
}
|
||||
|
||||
type MetadataRefreshRequester interface {
|
||||
RequestStaleMetadataRefresh(ctx context.Context, targetType, contentID string) error
|
||||
}
|
||||
@@ -102,6 +107,15 @@ func NewItemsHandler(
|
||||
}
|
||||
}
|
||||
|
||||
// SetCompletionObserver wires an optional observer notified when a watch
|
||||
// completes (used to auto-remove fully-watched items from the watchlist).
|
||||
func (h *ItemsHandler) SetCompletionObserver(obs watchstate.CompletionObserver) {
|
||||
if h == nil || h.watchState == nil || obs == nil {
|
||||
return
|
||||
}
|
||||
h.watchState.WithCompletionObserver(obs)
|
||||
}
|
||||
|
||||
// SetProfileStaler configures an optional staleness trigger for taste profiles.
|
||||
func (h *ItemsHandler) SetProfileStaler(ps ProfileStaler) {
|
||||
h.profileStaler = ps
|
||||
@@ -116,6 +130,13 @@ func (h *ItemsHandler) SetMetadataRefreshRequester(requester MetadataRefreshRequ
|
||||
h.metadataRefreshRequester = requester
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) SetCatalogSearchProvider(provider catalog.CatalogSearchProvider) {
|
||||
if h == nil || h.catalogResolver == nil || provider == nil {
|
||||
return
|
||||
}
|
||||
h.catalogResolver.WithSearchProvider(provider)
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) SetLocalWatchEventDispatcher(dispatcher LocalWatchEventDispatcher) {
|
||||
h.localWatchDispatcher = dispatcher
|
||||
}
|
||||
@@ -195,37 +216,42 @@ func (h *ItemsHandler) requestStaleMetadataRefresh(ctx context.Context, targetTy
|
||||
|
||||
// itemListResponse is the shape of a single item in browse/search list responses.
|
||||
type itemListResponse struct {
|
||||
ContentID string `json:"content_id"`
|
||||
Type string `json:"type"`
|
||||
Title string `json:"title"`
|
||||
SeriesTitle string `json:"series_title,omitempty"`
|
||||
SeasonNumber *int `json:"season_number,omitempty"`
|
||||
EpisodeNumber *int `json:"episode_number,omitempty"`
|
||||
Year int `json:"year,omitempty"`
|
||||
Runtime int `json:"runtime,omitempty"`
|
||||
Genres []string `json:"genres"`
|
||||
Keywords []string `json:"keywords"`
|
||||
Studios []string `json:"studios,omitempty"`
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
ContentRating string `json:"content_rating,omitempty"`
|
||||
Status string `json:"status"`
|
||||
ShowStatus string `json:"show_status,omitempty"`
|
||||
RatingIMDB *float64 `json:"rating_imdb,omitempty"`
|
||||
RatingTMDB *float64 `json:"rating_tmdb,omitempty"`
|
||||
RatingRTCritic *int `json:"rating_rt_critic,omitempty"`
|
||||
RatingRTAudience *int `json:"rating_rt_audience,omitempty"`
|
||||
OriginalLanguage string `json:"original_language,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
PosterURL string `json:"poster_url,omitempty"`
|
||||
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
|
||||
BackdropURL string `json:"backdrop_url,omitempty"`
|
||||
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
|
||||
ReleaseDate *string `json:"release_date,omitempty"`
|
||||
LastAirDate *string `json:"last_air_date,omitempty"`
|
||||
AddedAt *time.Time `json:"added_at,omitempty"`
|
||||
OverlaySummary *models.OverlaySummary `json:"overlay_summary,omitempty"`
|
||||
SortMetrics *sortMetricsResponse `json:"sort_metrics,omitempty"`
|
||||
UserState *itemUserStateResponse `json:"user_state,omitempty"`
|
||||
ContentID string `json:"content_id"`
|
||||
Type string `json:"type"`
|
||||
Title string `json:"title"`
|
||||
SeriesTitle string `json:"series_title,omitempty"`
|
||||
SeasonNumber *int `json:"season_number,omitempty"`
|
||||
EpisodeNumber *int `json:"episode_number,omitempty"`
|
||||
Year int `json:"year,omitempty"`
|
||||
Runtime int `json:"runtime,omitempty"`
|
||||
Genres []string `json:"genres"`
|
||||
Keywords []string `json:"keywords"`
|
||||
Studios []string `json:"studios,omitempty"`
|
||||
Networks []string `json:"networks,omitempty"`
|
||||
ContentRating string `json:"content_rating,omitempty"`
|
||||
Status string `json:"status"`
|
||||
ShowStatus string `json:"show_status,omitempty"`
|
||||
RatingIMDB *float64 `json:"rating_imdb,omitempty"`
|
||||
RatingTMDB *float64 `json:"rating_tmdb,omitempty"`
|
||||
RatingRTCritic *int `json:"rating_rt_critic,omitempty"`
|
||||
RatingRTAudience *int `json:"rating_rt_audience,omitempty"`
|
||||
OriginalLanguage string `json:"original_language,omitempty"`
|
||||
Overview string `json:"overview,omitempty"`
|
||||
PosterURL string `json:"poster_url,omitempty"`
|
||||
PosterThumbhash string `json:"poster_thumbhash,omitempty"`
|
||||
BackdropURL string `json:"backdrop_url,omitempty"`
|
||||
BackdropThumbhash string `json:"backdrop_thumbhash,omitempty"`
|
||||
ReleaseDate *string `json:"release_date,omitempty"`
|
||||
LastAirDate *string `json:"last_air_date,omitempty"`
|
||||
AddedAt *time.Time `json:"added_at,omitempty"`
|
||||
MangaChapterCount *int `json:"manga_chapter_count,omitempty"`
|
||||
MangaVolumeCount *int `json:"manga_volume_count,omitempty"`
|
||||
OverlaySummary *models.OverlaySummary `json:"overlay_summary,omitempty"`
|
||||
SortMetrics *sortMetricsResponse `json:"sort_metrics,omitempty"`
|
||||
UserState *itemUserStateResponse `json:"user_state,omitempty"`
|
||||
WorkID string `json:"work_id,omitempty"`
|
||||
WorkTitle string `json:"work_title,omitempty"`
|
||||
WorkFormats []catalog.WorkFormatSummary `json:"work_formats,omitempty"`
|
||||
}
|
||||
|
||||
type sortMetricsResponse struct {
|
||||
@@ -241,11 +267,17 @@ type sortMetricsResponse struct {
|
||||
SeriesName string `json:"series_name,omitempty"`
|
||||
}
|
||||
|
||||
type itemListImageURLs struct {
|
||||
posterURL string
|
||||
backdropURL string
|
||||
}
|
||||
|
||||
// browseResponse is the paginated response for the /items endpoint.
|
||||
type browseResponse struct {
|
||||
Total int `json:"total"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Items []itemListResponse `json:"items"`
|
||||
Total int `json:"total"`
|
||||
TotalExact bool `json:"total_exact"`
|
||||
HasMore bool `json:"has_more"`
|
||||
Items []itemListResponse `json:"items"`
|
||||
}
|
||||
|
||||
type itemFiltersResponse struct {
|
||||
@@ -387,7 +419,7 @@ func (h *ItemsHandler) HandleGetWatchDetail(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
}
|
||||
|
||||
if detail.Type == "movie" || detail.Type == "episode" || detail.Type == "ebook" {
|
||||
if detail.Type == "movie" || detail.Type == "episode" || detail.Type == "ebook" || detail.Type == "audiobook" {
|
||||
detail.UserData = h.getLeafUserData(r, detail.ContentID, detail.Type)
|
||||
applyEffectiveEditionPreference(detail.UserData, &detail.EffectiveVersionEditionKey)
|
||||
}
|
||||
@@ -578,9 +610,10 @@ func (h *ItemsHandler) writeCatalogBrowseResponse(w http.ResponseWriter, r *http
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, browseResponse{
|
||||
Total: result.Total,
|
||||
HasMore: result.HasMore,
|
||||
Items: items,
|
||||
Total: result.Total,
|
||||
TotalExact: result.TotalExact,
|
||||
HasMore: result.HasMore,
|
||||
Items: items,
|
||||
})
|
||||
return true
|
||||
}
|
||||
@@ -628,6 +661,13 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
|
||||
item = localized
|
||||
}
|
||||
}
|
||||
resp := itemListResponseShell(item, overlaySummary, userState)
|
||||
resp.PosterURL = h.presignURL(r, cardThumbnailPath(item.PosterPath), "card")
|
||||
resp.BackdropURL = h.presignURL(r, cardThumbnailPath(item.BackdropPath), "card")
|
||||
return resp
|
||||
}
|
||||
|
||||
func itemListResponseShell(item *models.MediaItem, overlaySummary *models.OverlaySummary, userState *itemUserStateResponse) itemListResponse {
|
||||
resp := itemListResponse{
|
||||
ContentID: item.ContentID,
|
||||
Type: item.Type,
|
||||
@@ -654,10 +694,10 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
|
||||
}
|
||||
|
||||
resp.AddedAt = item.AddedAt
|
||||
resp.MangaChapterCount = item.MangaChapterCount
|
||||
resp.MangaVolumeCount = item.MangaVolumeCount
|
||||
resp.ReleaseDate = item.ReleaseDate
|
||||
resp.LastAirDate = item.LastAirDate
|
||||
resp.PosterURL = h.presignURL(r, cardThumbnailPath(item.PosterPath), "card")
|
||||
resp.BackdropURL = h.presignURL(r, cardThumbnailPath(item.BackdropPath), "card")
|
||||
if resp.Genres == nil {
|
||||
resp.Genres = []string{}
|
||||
}
|
||||
@@ -668,6 +708,67 @@ func (h *ItemsHandler) toItemListResponseWithOverlay(r *http.Request, item *mode
|
||||
return resp
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) localizeItemListModels(ctx context.Context, items []*models.MediaItem, filter catalog.AccessFilter) []*models.MediaItem {
|
||||
if h == nil || h.detailSvc == nil || len(items) == 0 {
|
||||
return items
|
||||
}
|
||||
localized, err := h.detailSvc.LocalizeItemModels(ctx, items, filter)
|
||||
if err != nil || len(localized) != len(items) {
|
||||
return items
|
||||
}
|
||||
return localized
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) itemListCardImageURLs(ctx context.Context, items []*models.MediaItem) map[string]itemListImageURLs {
|
||||
urls := make(map[string]itemListImageURLs, len(items))
|
||||
if h == nil || h.detailSvc == nil || len(items) == 0 {
|
||||
return urls
|
||||
}
|
||||
|
||||
type pendingImages struct {
|
||||
contentID string
|
||||
posterPath string
|
||||
backdropPath string
|
||||
}
|
||||
|
||||
pending := make([]pendingImages, 0, len(items))
|
||||
paths := make([]string, 0, len(items)*2)
|
||||
seenPaths := make(map[string]struct{}, len(items)*2)
|
||||
addPath := func(path string) {
|
||||
if path == "" || path == "-" {
|
||||
return
|
||||
}
|
||||
if _, ok := seenPaths[path]; ok {
|
||||
return
|
||||
}
|
||||
seenPaths[path] = struct{}{}
|
||||
paths = append(paths, path)
|
||||
}
|
||||
|
||||
for _, item := range items {
|
||||
if item == nil || item.ContentID == "" {
|
||||
continue
|
||||
}
|
||||
images := pendingImages{
|
||||
contentID: item.ContentID,
|
||||
posterPath: cardThumbnailPath(item.PosterPath),
|
||||
backdropPath: cardThumbnailPath(item.BackdropPath),
|
||||
}
|
||||
pending = append(pending, images)
|
||||
addPath(images.posterPath)
|
||||
addPath(images.backdropPath)
|
||||
}
|
||||
|
||||
resolved := h.detailSvc.PresignURLsWithExpiry(ctx, paths, "card")
|
||||
for _, images := range pending {
|
||||
urls[images.contentID] = itemListImageURLs{
|
||||
posterURL: resolved[images.posterPath].URL,
|
||||
backdropURL: resolved[images.backdropPath].URL,
|
||||
}
|
||||
}
|
||||
return urls
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) listEpisodeBrowseMetadata(
|
||||
ctx context.Context,
|
||||
items []*models.MediaItem,
|
||||
@@ -954,7 +1055,7 @@ func (h *ItemsHandler) listOverlaySummaries(ctx context.Context, items []*models
|
||||
return summaries
|
||||
}
|
||||
|
||||
groupedFiles := h.listBrowseItemFiles(ctx, items, filter)
|
||||
groupedFiles := h.listBrowseItemOverlayFiles(ctx, items, filter)
|
||||
for contentID, files := range groupedFiles {
|
||||
if summary := overlays.BuildSummary(files); summary != nil {
|
||||
summaries[contentID] = summary
|
||||
@@ -1256,21 +1357,7 @@ func (h *ItemsHandler) listBrowseItemFiles(ctx context.Context, items []*models.
|
||||
|
||||
contentIDs := make([]string, 0, len(items))
|
||||
episodeIDs := make([]string, 0)
|
||||
seen := make(map[string]struct{}, len(items))
|
||||
for _, item := range items {
|
||||
if item == nil || item.ContentID == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[item.ContentID]; ok {
|
||||
continue
|
||||
}
|
||||
seen[item.ContentID] = struct{}{}
|
||||
if item.Type == "episode" {
|
||||
episodeIDs = append(episodeIDs, item.ContentID)
|
||||
} else {
|
||||
contentIDs = append(contentIDs, item.ContentID)
|
||||
}
|
||||
}
|
||||
collectBrowseFileIDs(items, &contentIDs, &episodeIDs)
|
||||
|
||||
if len(contentIDs) > 0 {
|
||||
contentFiles, err := h.fileRepo.ListByContentIDs(ctx, contentIDs)
|
||||
@@ -1295,6 +1382,60 @@ func (h *ItemsHandler) listBrowseItemFiles(ctx context.Context, items []*models.
|
||||
return grouped
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) listBrowseItemOverlayFiles(ctx context.Context, items []*models.MediaItem, filter catalog.AccessFilter) map[string][]*models.MediaFile {
|
||||
if h.fileRepo == nil || len(items) == 0 {
|
||||
return map[string][]*models.MediaFile{}
|
||||
}
|
||||
|
||||
overlayProvider, ok := h.fileRepo.(overlayFileProvider)
|
||||
if !ok {
|
||||
return h.listBrowseItemFiles(ctx, items, filter)
|
||||
}
|
||||
|
||||
contentIDs := make([]string, 0, len(items))
|
||||
episodeIDs := make([]string, 0)
|
||||
collectBrowseFileIDs(items, &contentIDs, &episodeIDs)
|
||||
|
||||
grouped := make(map[string][]*models.MediaFile, len(items))
|
||||
if len(contentIDs) > 0 {
|
||||
contentFiles, err := overlayProvider.ListOverlayFilesByContentIDs(ctx, contentIDs)
|
||||
if err != nil {
|
||||
return h.listBrowseItemFiles(ctx, items, filter)
|
||||
}
|
||||
for contentID, files := range contentFiles {
|
||||
grouped[contentID] = catalog.FilterMediaFilesByAccess(files, filter)
|
||||
}
|
||||
}
|
||||
if len(episodeIDs) > 0 {
|
||||
episodeFiles, err := overlayProvider.ListOverlayFilesByEpisodeIDs(ctx, episodeIDs)
|
||||
if err != nil {
|
||||
return h.listBrowseItemFiles(ctx, items, filter)
|
||||
}
|
||||
for episodeID, files := range episodeFiles {
|
||||
grouped[episodeID] = catalog.FilterMediaFilesByAccess(files, filter)
|
||||
}
|
||||
}
|
||||
return grouped
|
||||
}
|
||||
|
||||
func collectBrowseFileIDs(items []*models.MediaItem, contentIDs *[]string, episodeIDs *[]string) {
|
||||
seen := make(map[string]struct{}, len(items))
|
||||
for _, item := range items {
|
||||
if item == nil || item.ContentID == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[item.ContentID]; ok {
|
||||
continue
|
||||
}
|
||||
seen[item.ContentID] = struct{}{}
|
||||
if item.Type == "episode" {
|
||||
*episodeIDs = append(*episodeIDs, item.ContentID)
|
||||
} else {
|
||||
*contentIDs = append(*contentIDs, item.ContentID)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func firstNonBlankPtr(values ...*string) *string {
|
||||
for _, value := range values {
|
||||
if value == nil || strings.TrimSpace(*value) == "" {
|
||||
@@ -1318,12 +1459,22 @@ func maxFileBitrate(files []*models.MediaFile) int {
|
||||
|
||||
// toSeasonResponse converts a Season model to an API response.
|
||||
func (h *ItemsHandler) toSeasonResponse(r *http.Request, seriesID string, s *models.Season) seasonResponse {
|
||||
episodes, _ := h.episodeRepo.ListBySeason(r.Context(), seriesID, s.SeasonNumber)
|
||||
return h.toSeasonResponseFromEpisodes(r, seriesID, s, episodes, h.getAggregateUserData(r, episodes))
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) toSeasonResponseFromEpisodes(
|
||||
r *http.Request,
|
||||
seriesID string,
|
||||
s *models.Season,
|
||||
episodes []*models.Episode,
|
||||
userData *catalog.SeasonUserData,
|
||||
) seasonResponse {
|
||||
if h.detailSvc != nil {
|
||||
if localized, err := h.detailSvc.LocalizeSeasonModel(r.Context(), s, h.accessFilter(r)); err == nil && localized != nil {
|
||||
s = localized
|
||||
}
|
||||
}
|
||||
episodes, _ := h.episodeRepo.ListBySeason(r.Context(), seriesID, s.SeasonNumber)
|
||||
|
||||
resp := seasonResponse{
|
||||
ContentID: s.ContentID,
|
||||
@@ -1338,7 +1489,7 @@ func (h *ItemsHandler) toSeasonResponse(r *http.Request, seriesID string, s *mod
|
||||
resp.AirDate = s.AirDate.Format("2006-01-02")
|
||||
}
|
||||
resp.PosterURL = h.presignURL(r, featuredPosterPath(s.PosterPath), "featured")
|
||||
resp.UserData = h.getAggregateUserData(r, episodes)
|
||||
resp.UserData = userData
|
||||
|
||||
return resp
|
||||
}
|
||||
@@ -1353,12 +1504,11 @@ func (h *ItemsHandler) getLeafUserData(r *http.Request, contentID string, itemTy
|
||||
return nil
|
||||
}
|
||||
|
||||
progress, err := store.GetProgress(r.Context(), profileID, contentID)
|
||||
if err != nil || progress == nil {
|
||||
progress, err := userstore.GetProgressWithCompletedHistory(r.Context(), store, profileID, contentID)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
return leafUserDataFromProgress(*progress)
|
||||
return leafUserDataFromProgress(progress)
|
||||
}
|
||||
|
||||
// listLeafUserData batch-fetches watch progress for the given content IDs in a
|
||||
@@ -1369,19 +1519,23 @@ func (h *ItemsHandler) listLeafUserData(r *http.Request, contentIDs []string) ma
|
||||
return nil
|
||||
}
|
||||
|
||||
progressMap, err := store.ListProgressByMediaItems(r.Context(), profileID, contentIDs)
|
||||
progressMap, err := userstore.ListProgressWithCompletedHistory(r.Context(), store, profileID, contentIDs)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
result := make(map[string]*catalog.SeasonUserData, len(progressMap))
|
||||
for contentID, progress := range progressMap {
|
||||
result[contentID] = leafUserDataFromProgress(progress)
|
||||
progressCopy := progress
|
||||
result[contentID] = leafUserDataFromProgress(&progressCopy)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func leafUserDataFromProgress(progress userstore.WatchProgress) *catalog.SeasonUserData {
|
||||
func leafUserDataFromProgress(progress *userstore.WatchProgress) *catalog.SeasonUserData {
|
||||
if progress == nil {
|
||||
return nil
|
||||
}
|
||||
return &catalog.SeasonUserData{
|
||||
PositionSeconds: progress.PositionSeconds,
|
||||
DurationSeconds: progress.DurationSeconds,
|
||||
@@ -1446,29 +1600,71 @@ func (h *ItemsHandler) getAggregateUserData(r *http.Request, episodes []*models.
|
||||
return nil
|
||||
}
|
||||
|
||||
var watchedCount int
|
||||
var inProgressCount int
|
||||
for _, ep := range episodes {
|
||||
progress, progressErr := store.GetProgress(r.Context(), profileID, ep.ContentID)
|
||||
if progressErr != nil || progress == nil {
|
||||
continue
|
||||
}
|
||||
if progress.Completed {
|
||||
watchedCount++
|
||||
continue
|
||||
}
|
||||
if progress.PositionSeconds > 0 {
|
||||
inProgressCount++
|
||||
}
|
||||
progressMap, err := h.listProgressForEpisodeIDs(r.Context(), store, profileID, episodeContentIDs(episodes))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
return catalog.EpisodeRollupUserData(episodes, progressMap)
|
||||
}
|
||||
|
||||
unplayedCount := len(episodes) - watchedCount
|
||||
return &catalog.SeasonUserData{
|
||||
WatchedCount: watchedCount,
|
||||
UnplayedCount: unplayedCount,
|
||||
InProgressCount: inProgressCount,
|
||||
Played: watchedCount == len(episodes),
|
||||
func (h *ItemsHandler) progressMapForEpisodes(r *http.Request, episodes []*models.Episode) (map[string]userstore.WatchProgress, bool) {
|
||||
store, profileID, ok := h.userStoreForRequest(r)
|
||||
if !ok {
|
||||
return nil, false
|
||||
}
|
||||
episodeIDs := episodeContentIDs(episodes)
|
||||
progressMap, err := h.listProgressForEpisodeIDs(r.Context(), store, profileID, episodeIDs)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
return progressMap, true
|
||||
}
|
||||
|
||||
func (h *ItemsHandler) listProgressForEpisodeIDs(ctx context.Context, store userstore.UserStore, profileID string, episodeIDs []string) (map[string]userstore.WatchProgress, error) {
|
||||
const chunkSize = 500
|
||||
progressMap := make(map[string]userstore.WatchProgress, len(episodeIDs))
|
||||
for start := 0; start < len(episodeIDs); start += chunkSize {
|
||||
end := start + chunkSize
|
||||
if end > len(episodeIDs) {
|
||||
end = len(episodeIDs)
|
||||
}
|
||||
chunk, err := userstore.ListProgressWithCompletedHistory(ctx, store, profileID, episodeIDs[start:end])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for id, progress := range chunk {
|
||||
progressMap[id] = progress
|
||||
}
|
||||
}
|
||||
return progressMap, nil
|
||||
}
|
||||
|
||||
func episodeContentIDs(episodes []*models.Episode) []string {
|
||||
ids := make([]string, 0, len(episodes))
|
||||
seen := make(map[string]struct{}, len(episodes))
|
||||
for _, ep := range episodes {
|
||||
if ep == nil || ep.ContentID == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[ep.ContentID]; ok {
|
||||
continue
|
||||
}
|
||||
seen[ep.ContentID] = struct{}{}
|
||||
ids = append(ids, ep.ContentID)
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
func flattenEpisodeGroups(groups map[int][]*models.Episode) []*models.Episode {
|
||||
var total int
|
||||
for _, episodes := range groups {
|
||||
total += len(episodes)
|
||||
}
|
||||
flattened := make([]*models.Episode, 0, total)
|
||||
for _, episodes := range groups {
|
||||
flattened = append(flattened, episodes...)
|
||||
}
|
||||
return flattened
|
||||
}
|
||||
|
||||
// requestProfileID resolves the active profile for a request, falling back to
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
func TestBrowseResponseIncludesTotalExact(t *testing.T) {
|
||||
data, err := json.Marshal(browseResponse{
|
||||
Total: 3,
|
||||
TotalExact: true,
|
||||
HasMore: false,
|
||||
Items: []itemListResponse{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(data), `"total_exact":true`) {
|
||||
t.Fatalf("browse response missing total_exact: %s", data)
|
||||
}
|
||||
}
|
||||
|
||||
type countingItemListImageResolver struct {
|
||||
singleCalls int
|
||||
batchCalls int
|
||||
batchPaths []string
|
||||
}
|
||||
|
||||
func (r *countingItemListImageResolver) ResolveImageURL(_ context.Context, path string, variant string) string {
|
||||
r.singleCalls++
|
||||
return "single:" + variant + ":" + path
|
||||
}
|
||||
|
||||
func (r *countingItemListImageResolver) ResolveImageURLs(_ context.Context, paths []string, variant string) map[string]string {
|
||||
resolved := r.ResolveImageURLsWithExpiry(context.Background(), paths, variant)
|
||||
out := make(map[string]string, len(resolved))
|
||||
for path, value := range resolved {
|
||||
out[path] = value.URL
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (r *countingItemListImageResolver) ResolveImageURLWithExpiry(_ context.Context, path string, variant string) catalog.ResolvedImageURL {
|
||||
r.singleCalls++
|
||||
return catalog.ResolvedImageURL{URL: "single:" + variant + ":" + path}
|
||||
}
|
||||
|
||||
func (r *countingItemListImageResolver) ResolveImageURLsWithExpiry(_ context.Context, paths []string, variant string) map[string]catalog.ResolvedImageURL {
|
||||
r.batchCalls++
|
||||
r.batchPaths = append(r.batchPaths[:0], paths...)
|
||||
out := make(map[string]catalog.ResolvedImageURL, len(paths))
|
||||
for _, path := range paths {
|
||||
out[path] = catalog.ResolvedImageURL{URL: "batch:" + variant + ":" + path}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestItemListCardImageURLsUsesBatchResolver(t *testing.T) {
|
||||
resolver := &countingItemListImageResolver{}
|
||||
detailSvc := &catalog.DetailService{}
|
||||
detailSvc.SetImageResolver(resolver)
|
||||
handler := &ItemsHandler{detailSvc: detailSvc}
|
||||
|
||||
items := []*models.MediaItem{
|
||||
{
|
||||
ContentID: "movie-1",
|
||||
PosterPath: "plugin://poster-1/original.jpg",
|
||||
BackdropPath: "plugin://backdrop-1/original.jpg",
|
||||
},
|
||||
{
|
||||
ContentID: "movie-2",
|
||||
PosterPath: "https://cdn.example/poster-2.jpg",
|
||||
BackdropPath: "plugin://backdrop-1/original.jpg",
|
||||
},
|
||||
}
|
||||
|
||||
urls := handler.itemListCardImageURLs(context.Background(), items)
|
||||
|
||||
if resolver.singleCalls != 0 {
|
||||
t.Fatalf("single resolver calls = %d, want 0", resolver.singleCalls)
|
||||
}
|
||||
if resolver.batchCalls != 1 {
|
||||
t.Fatalf("batch resolver calls = %d, want 1", resolver.batchCalls)
|
||||
}
|
||||
if got := urls["movie-1"].posterURL; got != "batch:card:plugin://poster-1/original.jpg" {
|
||||
t.Fatalf("movie-1 poster URL = %q", got)
|
||||
}
|
||||
if got := urls["movie-1"].backdropURL; got != "batch:card:plugin://backdrop-1/original.jpg" {
|
||||
t.Fatalf("movie-1 backdrop URL = %q", got)
|
||||
}
|
||||
if got := urls["movie-2"].posterURL; got != "https://cdn.example/poster-2.jpg" {
|
||||
t.Fatalf("movie-2 poster URL = %q", got)
|
||||
}
|
||||
if got := len(resolver.batchPaths); got != 2 {
|
||||
t.Fatalf("batch resolver path count = %d, want 2", got)
|
||||
}
|
||||
}
|
||||
|
||||
type overlayFastPathFileRepo struct {
|
||||
fullContentCalls int
|
||||
overlayContentCalls int
|
||||
overlayEpisodeCalls int
|
||||
}
|
||||
|
||||
func (r *overlayFastPathFileRepo) GetByContentID(context.Context, string) ([]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *overlayFastPathFileRepo) GetByEpisodeID(context.Context, string) ([]*models.MediaFile, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *overlayFastPathFileRepo) ListByContentIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
|
||||
r.fullContentCalls++
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (r *overlayFastPathFileRepo) ListOverlayFilesByContentIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
|
||||
r.overlayContentCalls++
|
||||
return map[string][]*models.MediaFile{
|
||||
"movie-1": {
|
||||
{
|
||||
ContentID: "movie-1",
|
||||
Resolution: "4k",
|
||||
CodecVideo: "hevc",
|
||||
CodecAudio: "eac3",
|
||||
AudioChannels: 6,
|
||||
HDR: true,
|
||||
Container: "mkv",
|
||||
MediaFolderID: 1,
|
||||
AudioTracks: []models.AudioTrack{{Codec: "eac3", Channels: 6, Default: true}},
|
||||
SubtitleTracks: []models.SubtitleTrack{{Language: "en"}},
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (r *overlayFastPathFileRepo) ListOverlayFilesByEpisodeIDs(context.Context, []string) (map[string][]*models.MediaFile, error) {
|
||||
r.overlayEpisodeCalls++
|
||||
return map[string][]*models.MediaFile{
|
||||
"episode-1": {
|
||||
{
|
||||
EpisodeID: "episode-1",
|
||||
Resolution: "1080p",
|
||||
CodecVideo: "h264",
|
||||
Container: "mp4",
|
||||
MediaFolderID: 1,
|
||||
},
|
||||
},
|
||||
}, nil
|
||||
}
|
||||
|
||||
func TestListOverlaySummariesUsesOverlayFileProjection(t *testing.T) {
|
||||
repo := &overlayFastPathFileRepo{}
|
||||
handler := &ItemsHandler{fileRepo: repo}
|
||||
|
||||
summaries := handler.listOverlaySummaries(context.Background(), []*models.MediaItem{
|
||||
{ContentID: "movie-1", Type: "movie"},
|
||||
{ContentID: "episode-1", Type: "episode"},
|
||||
}, catalog.AccessFilter{})
|
||||
|
||||
if repo.fullContentCalls != 0 {
|
||||
t.Fatalf("full content file calls = %d, want 0", repo.fullContentCalls)
|
||||
}
|
||||
if repo.overlayContentCalls != 1 {
|
||||
t.Fatalf("overlay content calls = %d, want 1", repo.overlayContentCalls)
|
||||
}
|
||||
if repo.overlayEpisodeCalls != 1 {
|
||||
t.Fatalf("overlay episode calls = %d, want 1", repo.overlayEpisodeCalls)
|
||||
}
|
||||
if got := summaries["movie-1"].Resolution; got != "2160p" {
|
||||
t.Fatalf("movie overlay resolution = %q, want 2160p", got)
|
||||
}
|
||||
if got := summaries["movie-1"].VideoCodec; got != "H.265" {
|
||||
t.Fatalf("movie overlay video codec = %q, want H.265", got)
|
||||
}
|
||||
if got := summaries["episode-1"].VideoCodec; got != "H.264" {
|
||||
t.Fatalf("episode overlay video codec = %q, want H.264", got)
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,18 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/auth"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
func TestGetLeafUserDataUsesEbookReaderProgress(t *testing.T) {
|
||||
@@ -45,3 +51,168 @@ func TestGetLeafUserDataUsesEbookReaderProgress(t *testing.T) {
|
||||
t.Fatalf("completed ebook user data = %#v", complete)
|
||||
}
|
||||
}
|
||||
|
||||
// Audiobooks resolve user_data through the same watch-progress store as
|
||||
// movies/episodes (not the ebook reader store), so an in-progress audiobook
|
||||
// reports its saved position_seconds — which is what lets the player resume.
|
||||
func TestGetLeafUserDataReturnsAudiobookProgress(t *testing.T) {
|
||||
store := newPlaybackTestStore(t)
|
||||
if err := store.SetProgress(context.Background(), "profile-1", "audiobook-1", 1234, 5000, userstore.ProgressThresholds{}); err != nil {
|
||||
t.Fatalf("seed progress: %v", err)
|
||||
}
|
||||
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
|
||||
|
||||
req := httptest.NewRequest("GET", "/watch/audiobook-1", nil)
|
||||
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
|
||||
ctx = apimw.SetProfileID(ctx, "profile-1")
|
||||
req = req.WithContext(ctx)
|
||||
|
||||
progress := handler.getLeafUserData(req, "audiobook-1", "audiobook")
|
||||
if progress == nil {
|
||||
t.Fatal("audiobook user data = nil, want saved progress")
|
||||
}
|
||||
if progress.PositionSeconds != 1234 {
|
||||
t.Fatalf("PositionSeconds = %v, want 1234", progress.PositionSeconds)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetLeafUserDataUsesCompletedHistoryWhenProgressMissing(t *testing.T) {
|
||||
store := newPlaybackTestStore(t)
|
||||
addCompletedHistoryForUserDataTest(t, store, "movie-history-only")
|
||||
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
|
||||
req := authorizedUserDataRequest()
|
||||
|
||||
userData := handler.getLeafUserData(req, "movie-history-only", "movie")
|
||||
if userData == nil {
|
||||
t.Fatal("movie user data = nil, want history-backed watched state")
|
||||
}
|
||||
if !userData.Played {
|
||||
t.Fatalf("Played = false, want true from completed history")
|
||||
}
|
||||
if userData.PositionSeconds != 0 || userData.IsInProgress {
|
||||
t.Fatalf("history-only user data = %+v, want no resume position", userData)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetLeafUserDataPreservesResumeWhenCompletedHistoryExists(t *testing.T) {
|
||||
store := newPlaybackTestStore(t)
|
||||
if err := store.SetProgressAt(
|
||||
context.Background(),
|
||||
"profile-1",
|
||||
"movie-rewatch",
|
||||
600,
|
||||
7200,
|
||||
false,
|
||||
time.Date(2026, 5, 5, 12, 0, 0, 0, time.UTC),
|
||||
); err != nil {
|
||||
t.Fatalf("seed progress: %v", err)
|
||||
}
|
||||
addCompletedHistoryForUserDataTest(t, store, "movie-rewatch")
|
||||
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
|
||||
req := authorizedUserDataRequest()
|
||||
|
||||
userData := handler.getLeafUserData(req, "movie-rewatch", "movie")
|
||||
if userData == nil {
|
||||
t.Fatal("movie user data = nil, want progress-backed user data")
|
||||
}
|
||||
if !userData.Played {
|
||||
t.Fatalf("Played = false, want true from completed history")
|
||||
}
|
||||
if userData.PositionSeconds != 600 || userData.DurationSeconds != 7200 || !userData.IsInProgress {
|
||||
t.Fatalf("resume fields = %+v, want in-progress resume preserved", userData)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetAggregateUserDataReturnsNilWhenProgressBatchFails(t *testing.T) {
|
||||
store := &failingBatchProgressStore{}
|
||||
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
|
||||
|
||||
req := httptest.NewRequest("GET", "/series/series-1/seasons/1", nil)
|
||||
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
|
||||
ctx = apimw.SetProfileID(ctx, "profile-1")
|
||||
req = req.WithContext(ctx)
|
||||
|
||||
episodes := make([]*models.Episode, 501)
|
||||
for i := range episodes {
|
||||
episodes[i] = &models.Episode{ContentID: "episode-" + strconv.Itoa(i+1)}
|
||||
}
|
||||
|
||||
if userData := handler.getAggregateUserData(req, episodes); userData != nil {
|
||||
t.Fatalf("getAggregateUserData() = %#v, want nil after batch failure", userData)
|
||||
}
|
||||
if store.calls != 2 {
|
||||
t.Fatalf("ListProgressByMediaItems calls = %d, want 2", store.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGetAggregateUserDataCountsCompletedHistory(t *testing.T) {
|
||||
store := newPlaybackTestStore(t)
|
||||
if err := store.SetProgressAt(
|
||||
context.Background(),
|
||||
"profile-1",
|
||||
"episode-progress-complete",
|
||||
0,
|
||||
1800,
|
||||
true,
|
||||
time.Date(2026, 5, 5, 12, 0, 0, 0, time.UTC),
|
||||
); err != nil {
|
||||
t.Fatalf("seed progress: %v", err)
|
||||
}
|
||||
addCompletedHistoryForUserDataTest(t, store, "episode-history-complete")
|
||||
handler := &ItemsHandler{storeProvider: testUserStoreProvider{store: store}}
|
||||
req := authorizedUserDataRequest()
|
||||
|
||||
userData := handler.getAggregateUserData(req, []*models.Episode{
|
||||
{ContentID: "episode-progress-complete"},
|
||||
{ContentID: "episode-history-complete"},
|
||||
{ContentID: "episode-unplayed"},
|
||||
})
|
||||
if userData == nil {
|
||||
t.Fatal("aggregate user data = nil, want counts")
|
||||
}
|
||||
if userData.WatchedCount != 2 || userData.UnplayedCount != 1 || userData.Played {
|
||||
t.Fatalf("aggregate user data = %+v, want two watched and one unplayed", userData)
|
||||
}
|
||||
}
|
||||
|
||||
type failingBatchProgressStore struct {
|
||||
userstore.UserStore
|
||||
calls int
|
||||
}
|
||||
|
||||
func (s *failingBatchProgressStore) ListProgressByMediaItems(
|
||||
_ context.Context,
|
||||
_ string,
|
||||
mediaItemIDs []string,
|
||||
) (map[string]userstore.WatchProgress, error) {
|
||||
s.calls++
|
||||
if s.calls == 2 {
|
||||
return nil, errors.New("progress batch failed")
|
||||
}
|
||||
progress := make(map[string]userstore.WatchProgress, len(mediaItemIDs))
|
||||
for _, id := range mediaItemIDs {
|
||||
progress[id] = userstore.WatchProgress{MediaItemID: id, Completed: true}
|
||||
}
|
||||
return progress, nil
|
||||
}
|
||||
|
||||
func authorizedUserDataRequest() *http.Request {
|
||||
req := httptest.NewRequest("GET", "/items/movie-1", nil)
|
||||
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 1})
|
||||
ctx = apimw.SetProfileID(ctx, "profile-1")
|
||||
return req.WithContext(ctx)
|
||||
}
|
||||
|
||||
func addCompletedHistoryForUserDataTest(t *testing.T, store userstore.UserStore, mediaItemID string) {
|
||||
t.Helper()
|
||||
if err := store.AddHistory(context.Background(), userstore.WatchHistoryEntry{
|
||||
ProfileID: "profile-1",
|
||||
MediaItemID: mediaItemID,
|
||||
WatchedAt: "2026-05-04T12:00:00Z",
|
||||
DurationSeconds: 7200,
|
||||
Completed: true,
|
||||
Source: userstore.WatchHistorySourceTrakt,
|
||||
}); err != nil {
|
||||
t.Fatalf("seed completed history: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -687,6 +687,26 @@ func (h *LibraryHandler) HandleUpdateLibrary(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
}
|
||||
|
||||
// Re-fetch metadata when the library's metadata language changed, so
|
||||
// existing items adopt the new language instead of keeping the one
|
||||
// stamped at first match. Quick mode suffices: the refresh item lister
|
||||
// includes complete-but-language-mismatched items.
|
||||
if h.JobRepo != nil && !strings.EqualFold(strings.TrimSpace(oldFolder.MetadataLanguage), strings.TrimSpace(folder.MetadataLanguage)) {
|
||||
job, jobErr := h.JobRepo.CreateLibraryRefresh(r.Context(), currentAdminUserID(r), adminjob.LibraryRefreshRequest{
|
||||
LibraryID: folder.ID,
|
||||
LibraryName: folder.Name,
|
||||
Mode: adminjob.LibraryRefreshModeQuick,
|
||||
}, "Queued metadata refresh after library language change")
|
||||
if jobErr != nil {
|
||||
var conflict *adminjob.ActiveJobConflictError
|
||||
if !errors.As(jobErr, &conflict) {
|
||||
slog.Warn("queue language-change metadata refresh failed", "library_id", folder.ID, "error", jobErr)
|
||||
}
|
||||
} else {
|
||||
publishEventJob(r.Context(), h.EventsHub, "job.created", job)
|
||||
}
|
||||
}
|
||||
|
||||
// Rescan when paths have changed (folders added or removed).
|
||||
if req.Paths != nil && !slices.Equal(oldFolder.Paths, *req.Paths) {
|
||||
if h.ScanQueue != nil {
|
||||
@@ -2102,6 +2122,8 @@ func metadataContentLevelsForLibraryType(libraryType string) []string {
|
||||
return []string{"audiobook"}
|
||||
case "ebooks", "ebook":
|
||||
return []string{"ebook"}
|
||||
case "manga":
|
||||
return []string{"manga"}
|
||||
case "mixed":
|
||||
return []string{"movie", "series", "season", "episode", "audiobook", "ebook"}
|
||||
default:
|
||||
@@ -2493,10 +2515,15 @@ func (h *LibraryHandler) HandleListUnmatchedItems(w http.ResponseWriter, r *http
|
||||
)`
|
||||
}
|
||||
|
||||
// Manga chapters carry their series' match state; the chapter rows
|
||||
// themselves stay 'pending' and are resolved through the manga series,
|
||||
// so they must not surface as actionable unmatched items here.
|
||||
mangaChapterGuard := ` AND ` + catalog.MangaChapterExclusionWhere("mi")
|
||||
|
||||
countSQL := `
|
||||
SELECT COUNT(*)
|
||||
FROM media_items mi
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')`
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')` + mangaChapterGuard
|
||||
countSQL += filter
|
||||
|
||||
var total int
|
||||
@@ -2519,10 +2546,10 @@ func (h *LibraryHandler) HandleListUnmatchedItems(w http.ResponseWriter, r *http
|
||||
WHERE mil.content_id = mi.content_id
|
||||
LIMIT 1
|
||||
) lib ON true
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')%s
|
||||
WHERE mi.status IN ('unmatched', 'pending', 'ambiguous')%s%s
|
||||
ORDER BY mi.title ASC, mi.content_id ASC
|
||||
LIMIT $%d OFFSET $%d
|
||||
`, filter, len(filterArgs)+1, len(filterArgs)+2)
|
||||
`, mangaChapterGuard, filter, len(filterArgs)+1, len(filterArgs)+2)
|
||||
|
||||
rows, err := h.pool.Query(r.Context(), listSQL, listArgs...)
|
||||
if err != nil {
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/adminjob"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeAdminJobCreator struct {
|
||||
mu sync.Mutex
|
||||
refreshs []adminjob.LibraryRefreshRequest
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) Create(context.Context, adminjob.CreateJobInput) (*models.AdminJob, error) {
|
||||
return &models.AdminJob{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) CreateLibraryRefresh(_ context.Context, _ int, req adminjob.LibraryRefreshRequest, _ string) (*models.AdminJob, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.refreshs = append(f.refreshs, req)
|
||||
return &models.AdminJob{}, nil
|
||||
}
|
||||
|
||||
func (f *fakeAdminJobCreator) libraryRefreshes() []adminjob.LibraryRefreshRequest {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]adminjob.LibraryRefreshRequest(nil), f.refreshs...)
|
||||
}
|
||||
|
||||
// TestHandleUpdateLibraryLanguageChangeQueuesRefresh covers the trigger half
|
||||
// of issue #211: changing a library's metadata language must enqueue a
|
||||
// library metadata refresh so existing items re-fetch in the new language,
|
||||
// while an update that does not change the language must not enqueue one.
|
||||
func TestHandleUpdateLibraryLanguageChangeQueuesRefresh(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("SILO_TEST_DATABASE_URL is not set")
|
||||
}
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
var folderID int
|
||||
if err := pool.QueryRow(ctx, `
|
||||
INSERT INTO media_folders (type, name, enabled, metadata_language)
|
||||
VALUES ('movies', 'Lang Change Test', true, 'zh')
|
||||
RETURNING id
|
||||
`).Scan(&folderID); err != nil {
|
||||
t.Fatalf("seed folder: %v", err)
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_folders WHERE id = $1`, folderID)
|
||||
})
|
||||
|
||||
jobs := &fakeAdminJobCreator{}
|
||||
h := NewLibraryHandler(catalog.NewFolderRepository(pool), nil, nil, pool, nil)
|
||||
h.JobRepo = jobs
|
||||
|
||||
doUpdate := func(body string) *httptest.ResponseRecorder {
|
||||
req := httptest.NewRequest(http.MethodPut, "/libraries/"+strconv.Itoa(folderID), strings.NewReader(body))
|
||||
chiCtx := chi.NewRouteContext()
|
||||
chiCtx.URLParams.Add("id", strconv.Itoa(folderID))
|
||||
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, chiCtx))
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleUpdateLibrary(rec, req)
|
||||
return rec
|
||||
}
|
||||
|
||||
// An update that does not touch the language must not enqueue a refresh.
|
||||
if rec := doUpdate(`{"name":"Lang Change Test Renamed"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("rename update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := jobs.libraryRefreshes(); len(got) != 0 {
|
||||
t.Fatalf("refresh jobs after rename = %d, want 0", len(got))
|
||||
}
|
||||
|
||||
// Changing the metadata language must enqueue a refresh for this library.
|
||||
if rec := doUpdate(`{"metadata_language":"da"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("language update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
refreshes := jobs.libraryRefreshes()
|
||||
if len(refreshes) != 1 {
|
||||
t.Fatalf("refresh jobs after language change = %d, want 1", len(refreshes))
|
||||
}
|
||||
if refreshes[0].LibraryID != folderID {
|
||||
t.Errorf("refresh library id = %d, want %d", refreshes[0].LibraryID, folderID)
|
||||
}
|
||||
|
||||
// Re-submitting the same language is a no-op update and must not enqueue.
|
||||
if rec := doUpdate(`{"metadata_language":"da"}`); rec.Code != http.StatusOK {
|
||||
t.Fatalf("same-language update status = %d (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
if got := jobs.libraryRefreshes(); len(got) != 1 {
|
||||
t.Fatalf("refresh jobs after same-language update = %d, want still 1", len(got))
|
||||
}
|
||||
}
|
||||
@@ -14,6 +14,7 @@ func TestMetadataContentLevelsForLibraryTypeIncludesEbooks(t *testing.T) {
|
||||
{name: "plural ebooks", libraryType: "ebooks", want: []string{"ebook"}},
|
||||
{name: "singular ebook", libraryType: "ebook", want: []string{"ebook"}},
|
||||
{name: "mixed includes ebook", libraryType: "mixed", want: []string{"movie", "series", "season", "episode", "audiobook", "ebook"}},
|
||||
{name: "manga", libraryType: "manga", want: []string{"manga"}},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
// TestHandleListUnmatchedItemsExcludesMangaChapters verifies that manga
|
||||
// chapter rows (type='ebook' items linked into a series via manga_chapters)
|
||||
// do not appear in the admin Unmatched queue, while genuinely pending items
|
||||
// still do. Regression test for issue #204.
|
||||
func TestHandleListUnmatchedItemsExcludesMangaChapters(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("set SILO_TEST_DATABASE_URL to run DB-backed unmatched items handler test")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
pool, err := pgxpool.New(ctx, dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
suffix := time.Now().UnixNano()
|
||||
// Embed a unique per-run tag in every seeded title and scope the search
|
||||
// query to it, so concurrent or leftover rows in a shared test database
|
||||
// cannot skew the exact-total assertion below.
|
||||
tag := fmt.Sprintf("issue204-%d", suffix)
|
||||
seriesID := fmt.Sprintf("manga-series-%d", suffix)
|
||||
chapterID := fmt.Sprintf("manga-chapter-%d", suffix)
|
||||
pendingID := fmt.Sprintf("pending-ebook-%d", suffix)
|
||||
t.Cleanup(func() {
|
||||
_, _ = pool.Exec(ctx, `DELETE FROM media_items WHERE content_id = ANY($1)`,
|
||||
[]string{chapterID, seriesID, pendingID})
|
||||
})
|
||||
|
||||
for _, row := range []struct {
|
||||
id, typ, title, status string
|
||||
}{
|
||||
{seriesID, "manga", tag + " Series", "matched"},
|
||||
{chapterID, "ebook", tag + " Series c001", "pending"},
|
||||
{pendingID, "ebook", tag + " Plain Ebook", "pending"},
|
||||
} {
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO media_items (content_id, type, title, status, genres)
|
||||
VALUES ($1, $2, $3, $4, '{}'::text[])
|
||||
`, row.id, row.typ, row.title, row.status); err != nil {
|
||||
t.Fatalf("seed media item %s: %v", row.id, err)
|
||||
}
|
||||
}
|
||||
if _, err := pool.Exec(ctx, `
|
||||
INSERT INTO manga_chapters (chapter_content_id, series_content_id, chapter_index)
|
||||
VALUES ($1, $2, 1)
|
||||
`, chapterID, seriesID); err != nil {
|
||||
t.Fatalf("seed manga chapter link: %v", err)
|
||||
}
|
||||
|
||||
h := NewLibraryHandler(nil, nil, nil, pool, nil)
|
||||
req := httptest.NewRequest(http.MethodGet, "/libraries/unmatched-items?q="+url.QueryEscape(tag), nil)
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleListUnmatchedItems(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, want 200 (body: %s)", rec.Code, rec.Body.String())
|
||||
}
|
||||
var resp struct {
|
||||
Items []struct {
|
||||
ContentID string `json:"content_id"`
|
||||
} `json:"items"`
|
||||
Total int `json:"total"`
|
||||
}
|
||||
if err := json.Unmarshal(rec.Body.Bytes(), &resp); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
|
||||
var sawChapter, sawPending bool
|
||||
for _, item := range resp.Items {
|
||||
switch item.ContentID {
|
||||
case chapterID:
|
||||
sawChapter = true
|
||||
case pendingID:
|
||||
sawPending = true
|
||||
}
|
||||
}
|
||||
if !sawPending {
|
||||
t.Errorf("pending ebook %s missing from unmatched list", pendingID)
|
||||
}
|
||||
if sawChapter {
|
||||
t.Errorf("manga chapter %s should be excluded from unmatched list", chapterID)
|
||||
}
|
||||
if resp.Total != 1 {
|
||||
t.Errorf("total = %d, want 1 (chapter must be excluded from the count too)", resp.Total)
|
||||
}
|
||||
}
|
||||
@@ -795,7 +795,8 @@ func (h *LibraryCollectionHandler) HandleListLibraryUserCollections(w http.Respo
|
||||
}
|
||||
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
collections, err := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, libraryID)
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
collections, err := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, profileID, libraryID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load user collections")
|
||||
return
|
||||
@@ -833,6 +834,7 @@ func (h *LibraryCollectionHandler) HandleListLibraryCollections(w http.ResponseW
|
||||
}
|
||||
|
||||
userID := apimw.GetUserID(r.Context())
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
groups, err := h.GroupRepo.ListByLibrary(r.Context(), libraryID)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load groups")
|
||||
@@ -855,7 +857,7 @@ func (h *LibraryCollectionHandler) HandleListLibraryCollections(w http.ResponseW
|
||||
continue
|
||||
}
|
||||
if !userCollectionsLoaded {
|
||||
loadedUserCollections, loadErr := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, libraryID)
|
||||
loadedUserCollections, loadErr := usercollections.ListServerVisibleByLibrary(r.Context(), h.UserCollectionPool, userID, profileID, libraryID)
|
||||
if loadErr != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load user collections")
|
||||
return
|
||||
@@ -1564,6 +1566,7 @@ func (h *LibraryCollectionHandler) HandleSyncAdminCollection(w http.ResponseWrit
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
slog.Error("collection sync failed", "collection_id", collectionID, "error", err)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to sync collection")
|
||||
return
|
||||
}
|
||||
@@ -3131,12 +3134,12 @@ func (h *LibraryCollectionHandler) loadLiveCollectionItems(r *http.Request, coll
|
||||
|
||||
switch {
|
||||
case len(collection.LibraryIDs) > 0:
|
||||
def.LibraryIDs = intersectCollectionLibraryIDs(def.LibraryIDs, collection.LibraryIDs)
|
||||
def.LibraryIDs = catalog.IntersectCollectionLibraryIDs(def.LibraryIDs, collection.LibraryIDs)
|
||||
if len(def.LibraryIDs) == 0 {
|
||||
return []itemListResponse{}, nil
|
||||
}
|
||||
case collection.LibraryID > 0:
|
||||
def.LibraryIDs = intersectCollectionLibraryIDs(def.LibraryIDs, []int{collection.LibraryID})
|
||||
def.LibraryIDs = catalog.IntersectCollectionLibraryIDs(def.LibraryIDs, []int{collection.LibraryID})
|
||||
if len(def.LibraryIDs) == 0 {
|
||||
return []itemListResponse{}, nil
|
||||
}
|
||||
@@ -3175,33 +3178,8 @@ func (e smartCollectionQueryError) Unwrap() error {
|
||||
return e.err
|
||||
}
|
||||
|
||||
func intersectCollectionLibraryIDs(existing, required []int) []int {
|
||||
if len(required) == 0 {
|
||||
return append([]int(nil), existing...)
|
||||
}
|
||||
if len(existing) == 0 {
|
||||
return append([]int(nil), required...)
|
||||
}
|
||||
|
||||
allowed := make(map[int]struct{}, len(required))
|
||||
for _, id := range required {
|
||||
allowed[id] = struct{}{}
|
||||
}
|
||||
|
||||
result := make([]int, 0, len(existing))
|
||||
seen := make(map[int]struct{}, len(existing))
|
||||
for _, id := range existing {
|
||||
if _, ok := allowed[id]; !ok {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[id]; ok {
|
||||
continue
|
||||
}
|
||||
seen[id] = struct{}{}
|
||||
result = append(result, id)
|
||||
}
|
||||
return result
|
||||
}
|
||||
// intersectCollectionLibraryIDs moved to catalog.IntersectCollectionLibraryIDs
|
||||
// so the web API and the Jellyfin-compat resolver share one implementation.
|
||||
|
||||
func (h *LibraryCollectionHandler) toItemListResponse(r *http.Request, item *models.MediaItem) itemListResponse {
|
||||
resp := itemListResponse{
|
||||
|
||||
@@ -0,0 +1,182 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/literaryworks"
|
||||
)
|
||||
|
||||
type LiteraryWorkService interface {
|
||||
GetWork(ctx context.Context, workID string, filter catalog.AccessFilter) (*literaryworks.DetailResponse, error)
|
||||
ListCandidates(ctx context.Context, contentID string, limit int) ([]literaryworks.Candidate, error)
|
||||
LinkItems(ctx context.Context, workID string, contentIDs []string) (string, error)
|
||||
UnlinkItem(ctx context.Context, workID, contentID string) error
|
||||
ConfirmMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) (string, error)
|
||||
IgnoreMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) error
|
||||
}
|
||||
|
||||
type LiteraryWorkHandler struct {
|
||||
Service LiteraryWorkService
|
||||
}
|
||||
|
||||
type literaryWorkLinkRequest struct {
|
||||
WorkID string `json:"work_id"`
|
||||
ContentIDs []string `json:"content_ids"`
|
||||
}
|
||||
|
||||
type literaryWorkDecisionRequest struct {
|
||||
SourceContentID string `json:"source_content_id"`
|
||||
TargetContentID string `json:"target_content_id"`
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleGetWork(w http.ResponseWriter, r *http.Request) {
|
||||
workID := strings.TrimSpace(chi.URLParam(r, "work_id"))
|
||||
if workID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "work_id is required")
|
||||
return
|
||||
}
|
||||
if h == nil || h.Service == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
|
||||
return
|
||||
}
|
||||
resp, err := h.Service.GetWork(r.Context(), workID, requestAccessFilter(r))
|
||||
if err != nil {
|
||||
if errors.Is(err, literaryworks.ErrWorkNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Work not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load work")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, resp)
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleListCandidates(w http.ResponseWriter, r *http.Request) {
|
||||
contentID := strings.TrimSpace(chi.URLParam(r, "content_id"))
|
||||
if contentID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "content_id is required")
|
||||
return
|
||||
}
|
||||
if h == nil || h.Service == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
|
||||
return
|
||||
}
|
||||
limit := 20
|
||||
if raw := strings.TrimSpace(r.URL.Query().Get("limit")); raw != "" {
|
||||
if parsed, err := strconv.Atoi(raw); err == nil && parsed > 0 && parsed <= 100 {
|
||||
limit = parsed
|
||||
}
|
||||
}
|
||||
candidates, err := h.Service.ListCandidates(r.Context(), contentID, limit)
|
||||
if err != nil {
|
||||
if errors.Is(err, literaryworks.ErrWorkNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list work candidates")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]any{"candidates": candidates})
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleLinkItems(w http.ResponseWriter, r *http.Request) {
|
||||
if h == nil || h.Service == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
|
||||
return
|
||||
}
|
||||
var req literaryWorkLinkRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
workID, err := h.Service.LinkItems(r.Context(), req.WorkID, req.ContentIDs)
|
||||
if err != nil {
|
||||
if errors.Is(err, literaryworks.ErrWorkNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to link work items")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"work_id": workID})
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleUnlinkItem(w http.ResponseWriter, r *http.Request) {
|
||||
workID := strings.TrimSpace(chi.URLParam(r, "work_id"))
|
||||
contentID := strings.TrimSpace(chi.URLParam(r, "content_id"))
|
||||
if workID == "" || contentID == "" {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "work_id and content_id are required")
|
||||
return
|
||||
}
|
||||
if h == nil || h.Service == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
|
||||
return
|
||||
}
|
||||
if err := h.Service.UnlinkItem(r.Context(), workID, contentID); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to unlink work item")
|
||||
return
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleConfirmMatch(w http.ResponseWriter, r *http.Request) {
|
||||
h.handleDecision(w, r, true)
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) HandleIgnoreMatch(w http.ResponseWriter, r *http.Request) {
|
||||
h.handleDecision(w, r, false)
|
||||
}
|
||||
|
||||
func (h *LiteraryWorkHandler) handleDecision(w http.ResponseWriter, r *http.Request, confirm bool) {
|
||||
if h == nil || h.Service == nil {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Literary works are not configured")
|
||||
return
|
||||
}
|
||||
var req literaryWorkDecisionRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
req.SourceContentID = strings.TrimSpace(req.SourceContentID)
|
||||
req.TargetContentID = strings.TrimSpace(req.TargetContentID)
|
||||
if req.SourceContentID == "" || req.TargetContentID == "" || req.SourceContentID == req.TargetContentID {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "source_content_id and target_content_id are required")
|
||||
return
|
||||
}
|
||||
claims := apimw.GetClaims(r.Context())
|
||||
userID := 0
|
||||
if claims != nil {
|
||||
userID = claims.UserID
|
||||
}
|
||||
var (
|
||||
workID string
|
||||
err error
|
||||
)
|
||||
if confirm {
|
||||
workID, err = h.Service.ConfirmMatch(r.Context(), req.SourceContentID, req.TargetContentID, userID)
|
||||
} else {
|
||||
err = h.Service.IgnoreMatch(r.Context(), req.SourceContentID, req.TargetContentID, userID)
|
||||
}
|
||||
if err != nil {
|
||||
if errors.Is(err, literaryworks.ErrWorkNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to record work match decision")
|
||||
return
|
||||
}
|
||||
if confirm {
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok", "work_id": workID})
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/literaryworks"
|
||||
)
|
||||
|
||||
type fakeLiteraryWorkService struct {
|
||||
work *literaryworks.DetailResponse
|
||||
err error
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) GetWork(ctx context.Context, workID string, filter catalog.AccessFilter) (*literaryworks.DetailResponse, error) {
|
||||
return f.work, f.err
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) ListCandidates(ctx context.Context, contentID string, limit int) ([]literaryworks.Candidate, error) {
|
||||
return []literaryworks.Candidate{{SourceContentID: contentID, TargetContentID: "audio-1", Score: 0.9}}, f.err
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) LinkItems(ctx context.Context, workID string, contentIDs []string) (string, error) {
|
||||
if workID != "" {
|
||||
return workID, f.err
|
||||
}
|
||||
return "work-1", f.err
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) UnlinkItem(ctx context.Context, workID, contentID string) error {
|
||||
return f.err
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) ConfirmMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) (string, error) {
|
||||
return "work-1", f.err
|
||||
}
|
||||
|
||||
func (f fakeLiteraryWorkService) IgnoreMatch(ctx context.Context, sourceContentID, targetContentID string, userID int) error {
|
||||
return f.err
|
||||
}
|
||||
|
||||
func TestLiteraryWorkHandlerGetWork(t *testing.T) {
|
||||
handler := &LiteraryWorkHandler{Service: fakeLiteraryWorkService{
|
||||
work: &literaryworks.DetailResponse{
|
||||
WorkID: "work-1",
|
||||
WorkTitle: "Project Hail Mary",
|
||||
Formats: []literaryworks.FormatResponse{
|
||||
{Type: "ebook", ContentID: "ebook-1"},
|
||||
{Type: "audiobook", ContentID: "audio-1"},
|
||||
},
|
||||
},
|
||||
}}
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/v1/works/work-1", nil)
|
||||
rctx := chi.NewRouteContext()
|
||||
rctx.URLParams.Add("work_id", "work-1")
|
||||
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, rctx))
|
||||
rec := httptest.NewRecorder()
|
||||
|
||||
handler.HandleGetWork(rec, req)
|
||||
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d body=%s", rec.Code, rec.Body.String())
|
||||
}
|
||||
if !strings.Contains(rec.Body.String(), `"formats"`) || !strings.Contains(rec.Body.String(), `"audiobook"`) {
|
||||
t.Fatalf("body = %s, want work formats", rec.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,7 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"log/slog"
|
||||
@@ -13,8 +14,27 @@ import (
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/metadata/translation"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type metadataAIItemAccess interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.MediaItem, error)
|
||||
EnsureAccessible(ctx context.Context, contentID string, filter catalog.AccessFilter) error
|
||||
}
|
||||
|
||||
type metadataAISeasonLookup interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.Season, error)
|
||||
}
|
||||
|
||||
type metadataAIEpisodeLookup interface {
|
||||
GetByID(ctx context.Context, contentID string) (*models.Episode, error)
|
||||
}
|
||||
|
||||
type metadataAITarget struct {
|
||||
kind translation.TargetKind
|
||||
accessContentID string
|
||||
}
|
||||
|
||||
// MetadataAIHandler exposes AI translation of catalog descriptions into the
|
||||
// localization tables. The admin routes are mounted under the per-item
|
||||
// metadata curation guard; the on-view route is viewer-facing and enforces
|
||||
@@ -22,7 +42,11 @@ import (
|
||||
type MetadataAIHandler struct {
|
||||
service *translation.Service
|
||||
// ItemAccess authorizes the viewer-facing on-view route; nil disables it.
|
||||
ItemAccess *catalog.ItemRepository
|
||||
ItemAccess metadataAIItemAccess
|
||||
// SeasonLookup and EpisodeLookup let the viewer route resolve non-item
|
||||
// detail pages to the parent series for authorization.
|
||||
SeasonLookup metadataAISeasonLookup
|
||||
EpisodeLookup metadataAIEpisodeLookup
|
||||
}
|
||||
|
||||
// NewMetadataAIHandler creates a handler backed by the given service.
|
||||
@@ -82,7 +106,16 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
UserID: scope.UserID,
|
||||
ProfileID: scope.ProfileID,
|
||||
}
|
||||
if err := h.ItemAccess.EnsureAccessible(r.Context(), contentID, filter); err != nil {
|
||||
target, err := h.resolveTranslationTarget(r.Context(), contentID)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to authorize item")
|
||||
return
|
||||
}
|
||||
if err := h.ItemAccess.EnsureAccessible(r.Context(), target.accessContentID, filter); err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
@@ -96,7 +129,7 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
requestedBy = &userID
|
||||
}
|
||||
|
||||
job, err := h.service.RequestOnView(r.Context(), contentID, req.TargetLanguage, requestedBy)
|
||||
job, err := h.service.RequestOnView(r.Context(), target.kind, contentID, req.TargetLanguage, requestedBy)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, translation.ErrNotConfigured):
|
||||
@@ -115,6 +148,51 @@ func (h *MetadataAIHandler) HandleTranslateOnView(w http.ResponseWriter, r *http
|
||||
writeJSON(w, http.StatusAccepted, map[string]any{"job": job})
|
||||
}
|
||||
|
||||
func (h *MetadataAIHandler) resolveTranslationTarget(ctx context.Context, contentID string) (metadataAITarget, error) {
|
||||
if h.ItemAccess == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
item, err := h.ItemAccess.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if item == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetItem, accessContentID: contentID}, nil
|
||||
case !errors.Is(err, catalog.ErrItemNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
|
||||
if h.SeasonLookup != nil {
|
||||
season, err := h.SeasonLookup.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if season == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetSeason, accessContentID: season.SeriesID}, nil
|
||||
case !errors.Is(err, catalog.ErrSeasonNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
}
|
||||
|
||||
if h.EpisodeLookup != nil {
|
||||
episode, err := h.EpisodeLookup.GetByID(ctx, contentID)
|
||||
switch {
|
||||
case err == nil:
|
||||
if episode == nil {
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
return metadataAITarget{kind: translation.TargetEpisode, accessContentID: episode.SeriesID}, nil
|
||||
case !errors.Is(err, catalog.ErrEpisodeNotFound):
|
||||
return metadataAITarget{}, err
|
||||
}
|
||||
}
|
||||
|
||||
return metadataAITarget{}, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
type translateMetadataRequest struct {
|
||||
TargetLanguage string `json:"target_language"`
|
||||
IncludeChildren *bool `json:"include_children"` // default true
|
||||
@@ -134,9 +212,18 @@ func (h *MetadataAIHandler) HandleTranslate(w http.ResponseWriter, r *http.Reque
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "target_language is required")
|
||||
return
|
||||
}
|
||||
includeChildren := true
|
||||
target, err := h.resolveTranslationTarget(r.Context(), contentID)
|
||||
if err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Item not found")
|
||||
return
|
||||
}
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to resolve item")
|
||||
return
|
||||
}
|
||||
includeChildren := target.kind == translation.TargetItem
|
||||
if req.IncludeChildren != nil {
|
||||
includeChildren = *req.IncludeChildren
|
||||
includeChildren = *req.IncludeChildren && target.kind == translation.TargetItem
|
||||
}
|
||||
|
||||
var requestedBy *int
|
||||
@@ -145,7 +232,7 @@ func (h *MetadataAIHandler) HandleTranslate(w http.ResponseWriter, r *http.Reque
|
||||
}
|
||||
|
||||
job, err := h.service.Enqueue(r.Context(), translation.JobRequest{
|
||||
TargetKind: translation.TargetItem,
|
||||
TargetKind: target.kind,
|
||||
ContentID: contentID,
|
||||
TargetLanguage: req.TargetLanguage,
|
||||
IncludeChildren: includeChildren,
|
||||
|
||||
@@ -0,0 +1,124 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/metadata/translation"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeMetadataAIItemAccess struct {
|
||||
items map[string]*models.MediaItem
|
||||
ensureErr map[string]error
|
||||
checked []string
|
||||
}
|
||||
|
||||
func (f *fakeMetadataAIItemAccess) GetByID(_ context.Context, contentID string) (*models.MediaItem, error) {
|
||||
if item := f.items[contentID]; item != nil {
|
||||
return item, nil
|
||||
}
|
||||
return nil, catalog.ErrItemNotFound
|
||||
}
|
||||
|
||||
func (f *fakeMetadataAIItemAccess) EnsureAccessible(_ context.Context, contentID string, _ catalog.AccessFilter) error {
|
||||
f.checked = append(f.checked, contentID)
|
||||
return f.ensureErr[contentID]
|
||||
}
|
||||
|
||||
type fakeMetadataAISeasonLookup map[string]*models.Season
|
||||
|
||||
func (f fakeMetadataAISeasonLookup) GetByID(_ context.Context, contentID string) (*models.Season, error) {
|
||||
if season := f[contentID]; season != nil {
|
||||
return season, nil
|
||||
}
|
||||
return nil, catalog.ErrSeasonNotFound
|
||||
}
|
||||
|
||||
type fakeMetadataAIEpisodeLookup map[string]*models.Episode
|
||||
|
||||
func (f fakeMetadataAIEpisodeLookup) GetByID(_ context.Context, contentID string) (*models.Episode, error) {
|
||||
if episode := f[contentID]; episode != nil {
|
||||
return episode, nil
|
||||
}
|
||||
return nil, catalog.ErrEpisodeNotFound
|
||||
}
|
||||
|
||||
func TestMetadataAIResolveOnViewTarget(t *testing.T) {
|
||||
itemAccess := &fakeMetadataAIItemAccess{
|
||||
items: map[string]*models.MediaItem{
|
||||
"movie-1": {ContentID: "movie-1", Type: "movie"},
|
||||
},
|
||||
ensureErr: map[string]error{},
|
||||
}
|
||||
handler := &MetadataAIHandler{
|
||||
ItemAccess: itemAccess,
|
||||
SeasonLookup: fakeMetadataAISeasonLookup{
|
||||
"season-1": {ContentID: "season-1", SeriesID: "series-1"},
|
||||
},
|
||||
EpisodeLookup: fakeMetadataAIEpisodeLookup{
|
||||
"episode-1": {ContentID: "episode-1", SeriesID: "series-1"},
|
||||
},
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
name string
|
||||
contentID string
|
||||
wantKind translation.TargetKind
|
||||
wantAccess string
|
||||
}{
|
||||
{
|
||||
name: "item",
|
||||
contentID: "movie-1",
|
||||
wantKind: translation.TargetItem,
|
||||
wantAccess: "movie-1",
|
||||
},
|
||||
{
|
||||
name: "season",
|
||||
contentID: "season-1",
|
||||
wantKind: translation.TargetSeason,
|
||||
wantAccess: "series-1",
|
||||
},
|
||||
{
|
||||
name: "episode",
|
||||
contentID: "episode-1",
|
||||
wantKind: translation.TargetEpisode,
|
||||
wantAccess: "series-1",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
itemAccess.checked = nil
|
||||
got, err := handler.resolveTranslationTarget(context.Background(), tc.contentID)
|
||||
if err != nil {
|
||||
t.Fatalf("resolveTranslationTarget: %v", err)
|
||||
}
|
||||
if got.kind != tc.wantKind {
|
||||
t.Fatalf("kind = %s, want %s", got.kind, tc.wantKind)
|
||||
}
|
||||
if got.accessContentID != tc.wantAccess {
|
||||
t.Fatalf("accessContentID = %q, want %q", got.accessContentID, tc.wantAccess)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestMetadataAIResolveTranslationTargetReportsMissingContent(t *testing.T) {
|
||||
itemAccess := &fakeMetadataAIItemAccess{
|
||||
items: map[string]*models.MediaItem{},
|
||||
ensureErr: map[string]error{},
|
||||
}
|
||||
handler := &MetadataAIHandler{
|
||||
ItemAccess: itemAccess,
|
||||
SeasonLookup: fakeMetadataAISeasonLookup{},
|
||||
EpisodeLookup: fakeMetadataAIEpisodeLookup{},
|
||||
}
|
||||
|
||||
_, err := handler.resolveTranslationTarget(context.Background(), "missing")
|
||||
if !errors.Is(err, catalog.ErrItemNotFound) {
|
||||
t.Fatalf("err = %v, want %v", err, catalog.ErrItemNotFound)
|
||||
}
|
||||
}
|
||||
@@ -43,6 +43,8 @@ type notificationSyncResponse struct {
|
||||
UnreadCount int `json:"unread_count"`
|
||||
}
|
||||
|
||||
type notificationApplePushDisplayResponse = notifications.NotificationDisplay
|
||||
|
||||
type unreadCountResponse struct {
|
||||
Count int `json:"count"`
|
||||
}
|
||||
@@ -150,6 +152,29 @@ func (h *NotificationsHandler) HandleGet(w http.ResponseWriter, r *http.Request)
|
||||
writeJSON(w, http.StatusOK, h.system.PayloadForRow(r.Context(), *row))
|
||||
}
|
||||
|
||||
// HandleApplePushDisplay handles GET /notifications/push/apple/display/{delivery_id}.
|
||||
// It returns only compact display metadata for notification-service-extension
|
||||
// enrichment, scoped to the active profile.
|
||||
func (h *NotificationsHandler) HandleApplePushDisplay(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
id := chi.URLParam(r, "delivery_id")
|
||||
if id == "" {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Notification not found")
|
||||
return
|
||||
}
|
||||
row, err := h.system.Deliveries.GetByID(r.Context(), profileID, id)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load notification")
|
||||
return
|
||||
}
|
||||
if row == nil {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Notification not found")
|
||||
return
|
||||
}
|
||||
writeJSON(w, http.StatusOK, notificationApplePushDisplayResponse(notifications.BuildNotificationDisplay(*row)))
|
||||
}
|
||||
|
||||
// HandleUnreadCount handles GET /notifications/unread-count.
|
||||
func (h *NotificationsHandler) HandleUnreadCount(w http.ResponseWriter, r *http.Request) {
|
||||
profileID := apimw.GetProfileID(r.Context())
|
||||
@@ -318,9 +343,7 @@ type capabilityWebhooks struct {
|
||||
}
|
||||
|
||||
// HandleCapability handles GET /notifications/capability. Clients render
|
||||
// setup UI from this response instead of introspecting admin settings. Push
|
||||
// channels report unavailable until they ship
|
||||
// (docs/superpowers/plans/notifications/02-03).
|
||||
// setup UI from this response instead of introspecting admin settings.
|
||||
func (h *NotificationsHandler) HandleCapability(w http.ResponseWriter, r *http.Request) {
|
||||
webhooks := capabilityWebhooks{Available: false, MaxPerProfile: 0, SupportedTypes: []string{}}
|
||||
if h.system.Webhooks != nil && h.system.Settings.WebhooksEnabled(r.Context()) {
|
||||
@@ -364,9 +387,21 @@ func (h *NotificationsHandler) HandleCapability(w http.ResponseWriter, r *http.R
|
||||
DigestHour: h.system.Settings.DiscordDigestHour(r.Context()),
|
||||
}
|
||||
}
|
||||
applePush := capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}}
|
||||
// Like web push above, availability requires both the wiring (cipher/store)
|
||||
// and the admin delivery toggle: Available must mean "setup will actually
|
||||
// deliver", not "the server could store a token".
|
||||
if h.system.PushDevices != nil && h.system.PushDevices.Available() &&
|
||||
h.system.Settings.ApplePushDeliveryEnabled(r.Context()) {
|
||||
applePush = capabilityPush{
|
||||
Available: true,
|
||||
Provider: notifications.PushProviderSiloRelay,
|
||||
SupportedModes: []string{notifications.PushModePrivatePush, notifications.PushModeInAppOnly},
|
||||
}
|
||||
}
|
||||
writeJSON(w, http.StatusOK, capabilityResponse{
|
||||
InApp: capabilityInApp{Enabled: h.system.Settings.UIEnabled(r.Context())},
|
||||
ApplePush: capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}},
|
||||
ApplePush: applePush,
|
||||
AndroidPush: capabilityPush{Available: false, Provider: "off", SupportedModes: []string{"in_app_only"}},
|
||||
WebPush: webPush,
|
||||
Webhooks: webhooks,
|
||||
|
||||
@@ -0,0 +1,134 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func TestHandleApplePushDisplayDB(t *testing.T) {
|
||||
dsn := os.Getenv("SILO_TEST_DATABASE_URL")
|
||||
if dsn == "" {
|
||||
t.Skip("set SILO_TEST_DATABASE_URL to run DB-backed notification display handler test")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
config, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
t.Fatalf("parse db config: %v", err)
|
||||
}
|
||||
config.MaxConns = 1
|
||||
pool, err := pgxpool.NewWithConfig(ctx, config)
|
||||
if err != nil {
|
||||
t.Fatalf("connect db: %v", err)
|
||||
}
|
||||
t.Cleanup(pool.Close)
|
||||
|
||||
if _, err := pool.Exec(ctx, `
|
||||
CREATE TEMP TABLE notification_deliveries (
|
||||
id text PRIMARY KEY,
|
||||
release_event_id text,
|
||||
user_id integer NOT NULL,
|
||||
profile_id text NOT NULL,
|
||||
library_id integer,
|
||||
series_id text,
|
||||
episode_id text,
|
||||
type text NOT NULL,
|
||||
reason_flags jsonb NOT NULL,
|
||||
status text NOT NULL DEFAULT 'delivered',
|
||||
read_at timestamptz,
|
||||
delivered_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE TEMP TABLE episodes (
|
||||
content_id text PRIMARY KEY,
|
||||
title text,
|
||||
season_number integer,
|
||||
episode_number integer,
|
||||
overview text
|
||||
);
|
||||
CREATE TEMP TABLE media_items (
|
||||
content_id text PRIMARY KEY,
|
||||
title text,
|
||||
poster_path text,
|
||||
poster_thumbhash text,
|
||||
poster_source_path text,
|
||||
type text,
|
||||
year integer,
|
||||
overview text,
|
||||
genres text[],
|
||||
content_rating text,
|
||||
rating_imdb double precision,
|
||||
rating_tmdb double precision,
|
||||
imdb_id text,
|
||||
tmdb_id text,
|
||||
tvdb_id text
|
||||
);
|
||||
INSERT INTO media_items (content_id, title, type, genres)
|
||||
VALUES ('series-1', 'Severance', 'series', ARRAY[]::text[]);
|
||||
INSERT INTO episodes (content_id, title, season_number, episode_number)
|
||||
VALUES ('episode-1', 'Hello, Ms. Cobel', 2, 1);
|
||||
INSERT INTO notification_deliveries (
|
||||
id, release_event_id, user_id, profile_id, library_id, series_id, episode_id,
|
||||
type, reason_flags, status
|
||||
) VALUES (
|
||||
'delivery-1', 'event-1', 42, 'profile-1', 7, 'series-1', 'episode-1',
|
||||
'episode.available', '{"favorite":true}', 'delivered'
|
||||
);
|
||||
`); err != nil {
|
||||
t.Fatalf("seed temp tables: %v", err)
|
||||
}
|
||||
|
||||
handler := NewNotificationsHandler(¬ifications.System{
|
||||
Deliveries: notifications.NewDeliveryRepository(pool),
|
||||
}, nil)
|
||||
router := chi.NewRouter()
|
||||
router.Get("/notifications/push/apple/display/{delivery_id}", handler.HandleApplePushDisplay)
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/notifications/push/apple/display/delivery-1", nil)
|
||||
req = req.WithContext(apimw.SetProfileID(req.Context(), "profile-1"))
|
||||
rr := httptest.NewRecorder()
|
||||
router.ServeHTTP(rr, req)
|
||||
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if got := rr.Header().Get("Cache-Control"); got != "no-store" {
|
||||
t.Fatalf("Cache-Control = %q", got)
|
||||
}
|
||||
var response notifications.NotificationDisplay
|
||||
if err := json.NewDecoder(rr.Body).Decode(&response); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if response.DeliveryID != "delivery-1" ||
|
||||
response.Title != "The latest episode of Severance S02E01 just dropped!" ||
|
||||
response.Body != "Hello, Ms. Cobel" ||
|
||||
response.ThreadID != "series:series-1" ||
|
||||
response.Category != "episode_available" ||
|
||||
response.URL != "/item/episode-1" {
|
||||
t.Fatalf("response = %+v", response)
|
||||
}
|
||||
|
||||
req = httptest.NewRequest(http.MethodGet, "/notifications/push/apple/display/delivery-1", nil)
|
||||
req = req.WithContext(apimw.SetProfileID(req.Context(), "other-profile"))
|
||||
rr = httptest.NewRecorder()
|
||||
router.ServeHTTP(rr, req)
|
||||
if rr.Code != http.StatusNotFound {
|
||||
t.Fatalf("cross-profile status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
if got := rr.Header().Get("Cache-Control"); got != "no-store" {
|
||||
t.Fatalf("cross-profile Cache-Control = %q", got)
|
||||
}
|
||||
if strings.Contains(rr.Body.String(), "Severance") {
|
||||
t.Fatalf("cross-profile response leaked display metadata: %s", rr.Body.String())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
)
|
||||
|
||||
type applePushRegisterRequest struct {
|
||||
DeviceID string `json:"device_id"`
|
||||
APNsToken string `json:"apns_token"`
|
||||
APNsEnvironment string `json:"apns_environment"`
|
||||
APNsTopic string `json:"apns_topic"`
|
||||
PushMode string `json:"push_mode"`
|
||||
}
|
||||
|
||||
type applePushRegisterResponse struct {
|
||||
ID string `json:"id"`
|
||||
ServerDeviceID string `json:"server_device_id"`
|
||||
Enabled bool `json:"enabled"`
|
||||
PushMode string `json:"push_mode"`
|
||||
}
|
||||
|
||||
func (h *NotificationsHandler) pushDevices() *notifications.PushDeviceService {
|
||||
if h == nil || h.system == nil {
|
||||
return nil
|
||||
}
|
||||
return h.system.PushDevices
|
||||
}
|
||||
|
||||
// HandleRegisterApplePushDevice handles POST /devices/push/apple.
|
||||
func (h *NotificationsHandler) HandleRegisterApplePushDevice(w http.ResponseWriter, r *http.Request) {
|
||||
service := h.pushDevices()
|
||||
if service == nil || !service.Available() {
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push registration is not available")
|
||||
return
|
||||
}
|
||||
|
||||
var req applePushRegisterRequest
|
||||
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "Invalid request body")
|
||||
return
|
||||
}
|
||||
|
||||
device, err := service.RegisterApple(r.Context(), apimw.GetUserID(r.Context()), apimw.GetProfileID(r.Context()), notifications.ApplePushRegistrationInput{
|
||||
DeviceID: req.DeviceID,
|
||||
APNsToken: req.APNsToken,
|
||||
APNsEnvironment: req.APNsEnvironment,
|
||||
APNsTopic: req.APNsTopic,
|
||||
PushMode: req.PushMode,
|
||||
})
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, notifications.ErrPushDeviceInvalid):
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeviceUnsupported):
|
||||
writeError(w, http.StatusUnprocessableEntity, "unsupported_push_device", err.Error())
|
||||
case errors.Is(err, notifications.ErrPushDeviceUnavailable):
|
||||
writeError(w, http.StatusServiceUnavailable, "unavailable", "Apple push registration is not available")
|
||||
default:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to register Apple push device")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
writeJSON(w, http.StatusOK, applePushRegisterResponse{
|
||||
ID: device.ID,
|
||||
ServerDeviceID: device.ServerDeviceID,
|
||||
Enabled: device.Enabled,
|
||||
PushMode: device.PushMode,
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/auth"
|
||||
"github.com/Silo-Server/silo-server/internal/notifications"
|
||||
"github.com/Silo-Server/silo-server/internal/secret"
|
||||
)
|
||||
|
||||
type handlerPushStore struct {
|
||||
got notifications.ApplePushDeviceRegistration
|
||||
calls int
|
||||
device *notifications.PushDevice
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *handlerPushStore) UpsertApple(ctx context.Context, registration notifications.ApplePushDeviceRegistration, cipher *secret.Cipher) (*notifications.PushDevice, error) {
|
||||
f.calls++
|
||||
f.got = registration
|
||||
if f.err != nil {
|
||||
return nil, f.err
|
||||
}
|
||||
if f.device != nil {
|
||||
return f.device, nil
|
||||
}
|
||||
return ¬ifications.PushDevice{
|
||||
ID: "push-row",
|
||||
ServerDeviceID: "server-device",
|
||||
Enabled: true,
|
||||
PushMode: registration.PushMode,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func handlerPushCipher(t *testing.T) *secret.Cipher {
|
||||
t.Helper()
|
||||
cipher, err := secret.New([]byte("01234567890123456789012345678901"))
|
||||
if err != nil {
|
||||
t.Fatalf("new cipher: %v", err)
|
||||
}
|
||||
return cipher
|
||||
}
|
||||
|
||||
func newApplePushRequest(body string) *http.Request {
|
||||
req := httptest.NewRequest(http.MethodPost, "/api/v1/devices/push/apple", strings.NewReader(body))
|
||||
ctx := apimw.SetClaims(req.Context(), &auth.Claims{UserID: 42, Role: "user", TokenType: auth.TokenTypeAccess})
|
||||
ctx = apimw.SetProfileID(ctx, "profile-1")
|
||||
return req.WithContext(ctx)
|
||||
}
|
||||
|
||||
func TestHandleRegisterApplePushDevice(t *testing.T) {
|
||||
store := &handlerPushStore{}
|
||||
handler := NewNotificationsHandler(¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(store, handlerPushCipher(t)),
|
||||
}, nil)
|
||||
|
||||
body := `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"` + strings.Repeat("a", 64) + `",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"org.siloserver.silo",
|
||||
"push_mode":"private_push"
|
||||
}`
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleRegisterApplePushDevice(rr, newApplePushRequest(body))
|
||||
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var response applePushRegisterResponse
|
||||
if err := json.NewDecoder(rr.Body).Decode(&response); err != nil {
|
||||
t.Fatalf("decode response: %v", err)
|
||||
}
|
||||
if response.ID != "push-row" || response.ServerDeviceID != "server-device" || !response.Enabled || response.PushMode != notifications.PushModePrivatePush {
|
||||
t.Fatalf("unexpected response: %+v", response)
|
||||
}
|
||||
if store.calls != 1 {
|
||||
t.Fatalf("store calls = %d, want 1", store.calls)
|
||||
}
|
||||
if store.got.UserID != 42 || store.got.ProfileID != "profile-1" || store.got.DeviceID != "local-device" {
|
||||
t.Fatalf("unexpected stored registration: %+v", store.got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleRegisterApplePushDeviceErrors(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
system *notifications.System
|
||||
body string
|
||||
wantStatus int
|
||||
}{
|
||||
{
|
||||
name: "service unavailable",
|
||||
system: ¬ifications.System{},
|
||||
body: `{}`,
|
||||
wantStatus: http.StatusServiceUnavailable,
|
||||
},
|
||||
{
|
||||
name: "invalid json",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "invalid field",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"abcd",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"org.siloserver.silo",
|
||||
"push_mode":"private_push"
|
||||
}`,
|
||||
wantStatus: http.StatusBadRequest,
|
||||
},
|
||||
{
|
||||
name: "unsupported topic",
|
||||
system: ¬ifications.System{
|
||||
PushDevices: notifications.NewPushDeviceService(&handlerPushStore{}, handlerPushCipher(t)),
|
||||
},
|
||||
body: `{
|
||||
"device_id":"local-device",
|
||||
"apns_token":"` + strings.Repeat("a", 64) + `",
|
||||
"apns_environment":"production",
|
||||
"apns_topic":"com.example.app",
|
||||
"push_mode":"private_push"
|
||||
}`,
|
||||
wantStatus: http.StatusUnprocessableEntity,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
handler := NewNotificationsHandler(tt.system, nil)
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleRegisterApplePushDevice(rr, newApplePushRequest(tt.body))
|
||||
if rr.Code != tt.wantStatus {
|
||||
t.Fatalf("status = %d, want %d, body = %s", rr.Code, tt.wantStatus, rr.Body.String())
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
+606
-197
File diff suppressed because it is too large
Load Diff
@@ -13,7 +13,11 @@ type playbackCommandRecord struct {
|
||||
Name playback.CommandName
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *playback.Session) error {
|
||||
// stopPlaybackSession stops a session. userInitiated must be true only for a
|
||||
// genuine user/admin stop (the recipe card is then deleted); false for system
|
||||
// teardown such as an ffmpeg-exit cleanup, where the card is kept so the client
|
||||
// can reconstruct.
|
||||
func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *playback.Session, userInitiated bool) error {
|
||||
if h == nil || session == nil || session.ID == "" {
|
||||
return playback.ErrSessionNotFound
|
||||
}
|
||||
@@ -21,11 +25,11 @@ func (h *PlaybackHandler) stopPlaybackSession(ctx context.Context, session *play
|
||||
if err := h.sessionMgr.StopSession(session.ID); err != nil {
|
||||
return err
|
||||
}
|
||||
h.finalizeSessionStop(ctx, session, true, "stop")
|
||||
h.finalizeSessionStop(ctx, session, true, "stop", userInitiated)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID string) error {
|
||||
func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID string, userInitiated bool) error {
|
||||
if h == nil || sessionID == "" {
|
||||
return playback.ErrSessionNotFound
|
||||
}
|
||||
@@ -33,7 +37,7 @@ func (h *PlaybackHandler) stopPlaybackSessionByID(ctx context.Context, sessionID
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return h.stopPlaybackSession(ctx, session)
|
||||
return h.stopPlaybackSession(ctx, session, userInitiated)
|
||||
}
|
||||
|
||||
func (h *PlaybackHandler) abortPlaybackSession(ctx context.Context, session *playback.Session) error {
|
||||
|
||||
@@ -46,6 +46,10 @@ type playbackSessionRow struct {
|
||||
IsPaused bool `json:"is_paused"`
|
||||
HasPlaybackControl bool `json:"has_playback_control"`
|
||||
ClientIP string `json:"client_ip,omitempty"`
|
||||
ClientName string `json:"client_name,omitempty"`
|
||||
ClientVersion string `json:"client_version,omitempty"`
|
||||
ClientLabel string `json:"client_label,omitempty"`
|
||||
ClientUserAgent string `json:"client_user_agent,omitempty"`
|
||||
AudioTrackIndex int `json:"audio_track_index"`
|
||||
TranscodeAudio bool `json:"transcode_audio"`
|
||||
StreamBitrateKbps *int `json:"stream_bitrate_kbps"`
|
||||
@@ -149,6 +153,9 @@ func (l *PlaybackSessionsLoader) Load(
|
||||
COALESCE(s.is_paused, FALSE),
|
||||
COALESCE(s.has_websocket, FALSE),
|
||||
COALESCE(HOST(s.client_ip), ''),
|
||||
COALESCE(s.client_name, ''),
|
||||
COALESCE(s.client_version, ''),
|
||||
COALESCE(s.client_user_agent, ''),
|
||||
COALESCE(s.audio_track_index, 0),
|
||||
COALESCE(s.transcode_audio, FALSE),
|
||||
s.stream_bitrate_kbps,
|
||||
@@ -203,7 +210,8 @@ func (l *PlaybackSessionsLoader) Load(
|
||||
&s.MediaTitle, &s.MediaType, &s.SeriesName, &s.EpisodeName, &s.SeasonNumber, &s.EpisodeNumber,
|
||||
&posterPath,
|
||||
&s.PlayMethod, &s.ReportingNode, &s.NodeDisplayName, &s.FileDuration, &s.StartedAt, &s.UpdatedAt,
|
||||
&s.PositionSeconds, &s.IsPaused, &s.HasPlaybackControl, &s.ClientIP, &s.AudioTrackIndex, &s.TranscodeAudio, &streamBitrateKbps,
|
||||
&s.PositionSeconds, &s.IsPaused, &s.HasPlaybackControl, &s.ClientIP, &s.ClientName, &s.ClientVersion,
|
||||
&s.ClientUserAgent, &s.AudioTrackIndex, &s.TranscodeAudio, &streamBitrateKbps,
|
||||
&s.TranscodeNodeURL, &s.TargetResolution, &s.TargetVideoCodec, &s.TargetAudioCodec, &targetBitrateKbps,
|
||||
&s.TranscodeHWAccel, &s.SourceContainer, &sourceBitrateKbps, &s.SourceVideoCodec, &s.SourceVideoResolution,
|
||||
&s.SourceAudioCodec, &sourceAudioChannels, &audioTracksJSON, &s.RequestedVideoCodec, &s.RequestedVideoResolution,
|
||||
@@ -215,6 +223,7 @@ func (l *PlaybackSessionsLoader) Load(
|
||||
s.TargetBitrateKbps = targetBitrateKbps
|
||||
s.SourceBitrateKbps = sourceBitrateKbps
|
||||
s.SourceAudioChannels = sourceAudioChannels
|
||||
s.ClientLabel = playbackClientDisplayName(s.ClientName, s.ClientVersion, s.ClientUserAgent)
|
||||
enrichPlaybackSessionRow(&s, audioTracksJSON)
|
||||
sessions = append(sessions, s)
|
||||
}
|
||||
@@ -335,6 +344,141 @@ func firstNonEmptyValue(values ...string) string {
|
||||
return ""
|
||||
}
|
||||
|
||||
func playbackClientDisplayName(name, version, userAgent string) string {
|
||||
name = strings.TrimSpace(name)
|
||||
version = shortPlaybackClientVersion(version)
|
||||
if name != "" {
|
||||
if version != "" {
|
||||
return name + " " + version
|
||||
}
|
||||
return name
|
||||
}
|
||||
|
||||
userAgent = strings.TrimSpace(userAgent)
|
||||
if userAgent == "" {
|
||||
return ""
|
||||
}
|
||||
|
||||
rules := []struct {
|
||||
label string
|
||||
tokens []string
|
||||
versionTokens []string
|
||||
}{
|
||||
{label: "Infuse", tokens: []string{"infuse"}, versionTokens: []string{"infuse-direct", "infuse"}},
|
||||
{label: "Findroid", tokens: []string{"findroid"}, versionTokens: []string{"findroid"}},
|
||||
{label: "Streamyfin", tokens: []string{"streamyfin"}, versionTokens: []string{"streamyfin"}},
|
||||
{label: "Swiftfin", tokens: []string{"swiftfin"}, versionTokens: []string{"swiftfin"}},
|
||||
{label: "Jellyfin", tokens: []string{"jellyfin"}, versionTokens: []string{"jellyfin"}},
|
||||
{label: "JellyCon", tokens: []string{"jellycon"}, versionTokens: []string{"jellycon"}},
|
||||
{label: "Wholphin", tokens: []string{"wholphin"}, versionTokens: []string{"wholphin"}},
|
||||
{label: "Fladder", tokens: []string{"fladder"}, versionTokens: []string{"fladder"}},
|
||||
{label: "VidHub", tokens: []string{"vidhub"}, versionTokens: []string{"vidhub"}},
|
||||
{label: "SenPlayer", tokens: []string{"senplayer"}, versionTokens: []string{"senplayer"}},
|
||||
{label: "Kodi", tokens: []string{"kodi"}, versionTokens: []string{"kodi"}},
|
||||
{label: "MPV", tokens: []string{"mpv"}, versionTokens: []string{"mpv"}},
|
||||
{label: "Edge", tokens: []string{"edg/"}, versionTokens: []string{"edg"}},
|
||||
{label: "Opera", tokens: []string{"opr/", "opera"}, versionTokens: []string{"opr", "opera"}},
|
||||
{label: "Firefox", tokens: []string{"firefox/", "fxios/"}, versionTokens: []string{"firefox", "fxios"}},
|
||||
{label: "Chrome", tokens: []string{"chrome/", "crios/"}, versionTokens: []string{"chrome", "crios"}},
|
||||
{label: "Safari", tokens: []string{"safari/"}, versionTokens: []string{"version"}},
|
||||
}
|
||||
lower := strings.ToLower(userAgent)
|
||||
for _, rule := range rules {
|
||||
if !containsAny(lower, rule.tokens) {
|
||||
continue
|
||||
}
|
||||
if version := firstProductVersion(userAgent, rule.versionTokens); version != "" {
|
||||
return rule.label + " " + version
|
||||
}
|
||||
return rule.label
|
||||
}
|
||||
|
||||
switch {
|
||||
case strings.Contains(lower, "applecoremedia"):
|
||||
return "Apple player"
|
||||
case strings.Contains(lower, "okhttp"):
|
||||
return "Android client"
|
||||
case strings.Contains(lower, "dart/"):
|
||||
return "Flutter client"
|
||||
case strings.Contains(lower, "go-http-client"):
|
||||
return "Go client"
|
||||
case strings.Contains(lower, "curl/"):
|
||||
return "curl"
|
||||
case strings.Contains(lower, "python-requests"):
|
||||
return "Python requests"
|
||||
default:
|
||||
return firstUserAgentProduct(userAgent)
|
||||
}
|
||||
}
|
||||
|
||||
func containsAny(value string, needles []string) bool {
|
||||
for _, needle := range needles {
|
||||
if strings.Contains(value, needle) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func firstProductVersion(userAgent string, tokens []string) string {
|
||||
for _, field := range strings.Fields(userAgent) {
|
||||
name, version, ok := strings.Cut(field, "/")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
name = strings.TrimSpace(strings.ToLower(name))
|
||||
for _, token := range tokens {
|
||||
if name == strings.TrimSpace(strings.ToLower(token)) {
|
||||
return shortPlaybackClientVersion(version)
|
||||
}
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func shortPlaybackClientVersion(version string) string {
|
||||
version = strings.Trim(strings.TrimSpace(version), `";),`)
|
||||
if version == "" {
|
||||
return ""
|
||||
}
|
||||
version = strings.Map(func(r rune) rune {
|
||||
if (r >= '0' && r <= '9') || r == '.' {
|
||||
return r
|
||||
}
|
||||
return -1
|
||||
}, version)
|
||||
parts := strings.Split(version, ".")
|
||||
filtered := parts[:0]
|
||||
for _, part := range parts {
|
||||
if part != "" {
|
||||
filtered = append(filtered, part)
|
||||
}
|
||||
}
|
||||
if len(filtered) == 0 {
|
||||
return ""
|
||||
}
|
||||
if len(filtered) > 2 {
|
||||
filtered = filtered[:2]
|
||||
}
|
||||
if len(filtered) == 2 && filtered[1] == "0" {
|
||||
filtered = filtered[:1]
|
||||
}
|
||||
return strings.Join(filtered, ".")
|
||||
}
|
||||
|
||||
func firstUserAgentProduct(userAgent string) string {
|
||||
for _, field := range strings.Fields(userAgent) {
|
||||
name, _, ok := strings.Cut(field, "/")
|
||||
if ok {
|
||||
name = strings.Trim(name, `";(),`)
|
||||
if name != "" && !strings.EqualFold(name, "Mozilla") {
|
||||
return name
|
||||
}
|
||||
}
|
||||
}
|
||||
return "Unknown client"
|
||||
}
|
||||
|
||||
func (l *PlaybackSessionsLoader) populateProfileNames(ctx context.Context, sessions []playbackSessionRow) {
|
||||
if l == nil || l.storeProv == nil || len(sessions) == 0 {
|
||||
return
|
||||
|
||||
@@ -22,6 +22,7 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/nodepool"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
"github.com/Silo-Server/silo-server/internal/streamtoken"
|
||||
"github.com/Silo-Server/silo-server/internal/transcodenode"
|
||||
"github.com/Silo-Server/silo-server/internal/userdb"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
@@ -1162,6 +1163,244 @@ func TestHandleStartTranscode_PreservesRecomputedBaseMethodAfterFallback(t *test
|
||||
}
|
||||
}
|
||||
|
||||
// newRemoteAudioSwitchSession builds a session that is already an offloaded
|
||||
// (remote) transcode with a populated target recipe, mirroring the post-start
|
||||
// state HandleChangeAudioTrack sees for an offloaded transcode.
|
||||
func newRemoteAudioSwitchSession(t *testing.T, sessionMgr *playback.SessionManager, file *models.MediaFile, nodeURL string) string {
|
||||
t.Helper()
|
||||
session, err := sessionMgr.StartSession(1, "profile-1", file.ID, playback.PlayTranscode, true)
|
||||
if err != nil {
|
||||
t.Fatalf("StartSession: %v", err)
|
||||
}
|
||||
if err := sessionMgr.UpdateStreamState(session.ID, playback.SessionStreamState{
|
||||
PlayMethod: playback.PlayTranscode,
|
||||
BasePlayMethod: playback.PlayTranscode,
|
||||
AudioTrackIndex: 0,
|
||||
TranscodeAudio: true,
|
||||
TargetResolution: "720p",
|
||||
TargetVideoCodec: "h264",
|
||||
TargetAudioCodec: "aac",
|
||||
TargetBitrateKbps: 2000,
|
||||
TranscodeHWAccel: "none",
|
||||
}); err != nil {
|
||||
t.Fatalf("UpdateStreamState: %v", err)
|
||||
}
|
||||
if err := sessionMgr.SetTranscodeNodeURL(session.ID, nodeURL); err != nil {
|
||||
t.Fatalf("SetTranscodeNodeURL: %v", err)
|
||||
}
|
||||
return session.ID
|
||||
}
|
||||
|
||||
// TestHandleChangeAudioTrack_RemoteTranscodeRestartsNodeAndMintsFullRecipe
|
||||
// verifies BUG A + BUG B: an audio switch on an offloaded transcode POSTs a
|
||||
// fresh /transcode/start to the node carrying the NEW AudioTrackIndex (so the
|
||||
// node's ffmpeg actually switches tracks), and the returned proxy URL carries a
|
||||
// recipe-COMPLETE stream token (not identity-only) so a later node restart can
|
||||
// reconstruct with the switched audio.
|
||||
func TestHandleChangeAudioTrack_RemoteTranscodeRestartsNodeAndMintsFullRecipe(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: writePlaybackTestMediaFile(t, "movie.mkv"),
|
||||
Resolution: "1080p",
|
||||
CodecVideo: "h264",
|
||||
CodecAudio: "aac",
|
||||
Container: "mkv",
|
||||
Bitrate: 8000,
|
||||
Duration: 3600,
|
||||
AudioTracks: []models.AudioTrack{
|
||||
{Codec: "aac", Default: true},
|
||||
{Codec: "ac3"},
|
||||
},
|
||||
}
|
||||
|
||||
var remoteStartRequests int
|
||||
var remoteStartReq transcodenode.TranscodeStartRequest
|
||||
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method != http.MethodPost || r.URL.Path != "/transcode/start" {
|
||||
t.Errorf("unexpected remote request %s %s", r.Method, r.URL.Path)
|
||||
http.Error(w, "unexpected", http.StatusBadRequest)
|
||||
return
|
||||
}
|
||||
remoteStartRequests++
|
||||
if err := json.NewDecoder(r.Body).Decode(&remoteStartReq); err != nil {
|
||||
t.Errorf("decode remote start request: %v", err)
|
||||
}
|
||||
w.WriteHeader(http.StatusAccepted)
|
||||
_ = json.NewEncoder(w).Encode(transcodenode.TranscodeStartResponse{
|
||||
SessionID: remoteStartReq.SessionID,
|
||||
Status: "accepted",
|
||||
HWAccel: "none",
|
||||
})
|
||||
}))
|
||||
defer remote.Close()
|
||||
|
||||
handler := NewPlaybackHandler(sessionMgr, testPlaybackFileResolver{file: file})
|
||||
handler.ItemAccess = allowAllPlaybackItemAccess{}
|
||||
handler.JWTSecret = "test-secret"
|
||||
|
||||
group := "rack-a"
|
||||
proxies := nodepool.NewProxyPool()
|
||||
proxies.SetNodes([]*nodepool.Node{{
|
||||
ID: 1, Name: "proxy-1", Type: nodepool.NodeTypeProxy, URL: "http://proxy-1",
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
transcodes := nodepool.NewTranscodePool()
|
||||
transcodes.SetNodes([]*nodepool.Node{{
|
||||
ID: 2, Name: "transcode-1", Type: nodepool.NodeTypeTranscode, URL: remote.URL,
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
handler.NodePlanner = nodepool.NewPlanner(proxies, transcodes)
|
||||
|
||||
sessionID := newRemoteAudioSwitchSession(t, sessionMgr, file, remote.URL)
|
||||
|
||||
changeReq := httptest.NewRequest(
|
||||
http.MethodPatch,
|
||||
"/api/v1/playback/"+sessionID+"/audio",
|
||||
strings.NewReader(`{"audio_track_index":1,"position":120}`),
|
||||
)
|
||||
changeReq = changeReq.WithContext(newAuthorizedPlaybackContext())
|
||||
changeReq = withPlaybackRouteParam(changeReq, "session_id", sessionID)
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleChangeAudioTrack(rr, changeReq)
|
||||
if rr.Code != http.StatusOK {
|
||||
t.Fatalf("change status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
|
||||
// BUG A: the node must have been told to restart on the new track.
|
||||
if remoteStartRequests != 1 {
|
||||
t.Fatalf("remote /transcode/start calls = %d, want 1", remoteStartRequests)
|
||||
}
|
||||
if remoteStartReq.AudioTrackIndex != 1 {
|
||||
t.Fatalf("remote AudioTrackIndex = %d, want 1", remoteStartReq.AudioTrackIndex)
|
||||
}
|
||||
if remoteStartReq.SessionID != sessionID {
|
||||
t.Fatalf("remote SessionID = %q, want %q", remoteStartReq.SessionID, sessionID)
|
||||
}
|
||||
// Seek 120s with the node-default 2s segments => start segment 60.
|
||||
if remoteStartReq.StartSegmentNumber != 60 {
|
||||
t.Fatalf("remote StartSegmentNumber = %d, want 60", remoteStartReq.StartSegmentNumber)
|
||||
}
|
||||
if remoteStartReq.TargetResolution != "720p" || remoteStartReq.TargetCodecVideo != "h264" {
|
||||
t.Fatalf("remote target recipe = %q/%q, want 720p/h264", remoteStartReq.TargetResolution, remoteStartReq.TargetCodecVideo)
|
||||
}
|
||||
|
||||
var resp changeAudioResponse
|
||||
if err := json.NewDecoder(rr.Body).Decode(&resp); err != nil {
|
||||
t.Fatalf("decode change response: %v", err)
|
||||
}
|
||||
|
||||
// BUG B: the returned proxy URL token must be recipe-COMPLETE.
|
||||
prefix := "http://proxy-1/stream/transcode/"
|
||||
suffix := "/master.m3u8"
|
||||
if !strings.HasPrefix(resp.StreamURL, prefix) || !strings.HasSuffix(resp.StreamURL, suffix) {
|
||||
t.Fatalf("stream URL = %q, want proxy transcode manifest URL", resp.StreamURL)
|
||||
}
|
||||
token := strings.TrimSuffix(strings.TrimPrefix(resp.StreamURL, prefix), suffix)
|
||||
claims, err := streamtoken.Verify(token, handler.JWTSecret)
|
||||
if err != nil {
|
||||
t.Fatalf("verify stream token: %v", err)
|
||||
}
|
||||
if claims.AudioTrackIndex != 1 {
|
||||
t.Fatalf("token AudioTrackIndex = %d, want 1", claims.AudioTrackIndex)
|
||||
}
|
||||
// Identity-only claims would leave these zero; a full recipe carries them.
|
||||
if claims.TargetCodec != "h264" {
|
||||
t.Fatalf("token TargetCodec (video) = %q, want h264 (recipe-complete)", claims.TargetCodec)
|
||||
}
|
||||
if claims.TargetCodecAudio != "aac" {
|
||||
t.Fatalf("token TargetCodecAudio = %q, want aac (recipe-complete)", claims.TargetCodecAudio)
|
||||
}
|
||||
if claims.TargetRes != "720p" {
|
||||
t.Fatalf("token TargetRes = %q, want 720p (recipe-complete)", claims.TargetRes)
|
||||
}
|
||||
if claims.TargetBitrateKbps != 2000 {
|
||||
t.Fatalf("token TargetBitrateKbps = %d, want 2000 (recipe-complete)", claims.TargetBitrateKbps)
|
||||
}
|
||||
if claims.SeekSeconds != 120 {
|
||||
t.Fatalf("token SeekSeconds = %v, want 120", claims.SeekSeconds)
|
||||
}
|
||||
if claims.StartSegmentNumber != 60 {
|
||||
t.Fatalf("token StartSegmentNumber = %d, want 60", claims.StartSegmentNumber)
|
||||
}
|
||||
// SegmentDuration must be > 0: the node treats a zero/omitted value as an
|
||||
// incomplete token and falls back to a recipe store the native path never
|
||||
// populates, which would 404 on a node restart. This guards that regression.
|
||||
if claims.SegmentDuration != playback.DefaultSegmentDuration {
|
||||
t.Fatalf("token SegmentDuration = %d, want %d (recipe-complete)", claims.SegmentDuration, playback.DefaultSegmentDuration)
|
||||
}
|
||||
if claims.MediaPath != file.FilePath {
|
||||
t.Fatalf("token MediaPath = %q, want %q", claims.MediaPath, file.FilePath)
|
||||
}
|
||||
if claims.TranscodeNode != remote.URL {
|
||||
t.Fatalf("token TranscodeNode = %q, want %q", claims.TranscodeNode, remote.URL)
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleChangeAudioTrack_RemoteTranscodeNodeUnreachableSurfacesError
|
||||
// verifies the chosen error posture: when the transcode node cannot be reached
|
||||
// for the restart, the handler surfaces a 502 rather than returning a 200 that
|
||||
// clients (silo-android / silo-apple) would trust as "audio switched".
|
||||
func TestHandleChangeAudioTrack_RemoteTranscodeNodeUnreachableSurfacesError(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: writePlaybackTestMediaFile(t, "movie.mkv"),
|
||||
Resolution: "1080p",
|
||||
CodecVideo: "h264",
|
||||
CodecAudio: "aac",
|
||||
Container: "mkv",
|
||||
Bitrate: 8000,
|
||||
Duration: 3600,
|
||||
AudioTracks: []models.AudioTrack{
|
||||
{Codec: "aac", Default: true},
|
||||
{Codec: "ac3"},
|
||||
},
|
||||
}
|
||||
|
||||
// A node URL that nothing is listening on => connection refused.
|
||||
deadNodeURL := "http://127.0.0.1:1"
|
||||
|
||||
handler := NewPlaybackHandler(sessionMgr, testPlaybackFileResolver{file: file})
|
||||
handler.ItemAccess = allowAllPlaybackItemAccess{}
|
||||
handler.JWTSecret = "test-secret"
|
||||
|
||||
group := "rack-a"
|
||||
proxies := nodepool.NewProxyPool()
|
||||
proxies.SetNodes([]*nodepool.Node{{
|
||||
ID: 1, Name: "proxy-1", Type: nodepool.NodeTypeProxy, URL: "http://proxy-1",
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
transcodes := nodepool.NewTranscodePool()
|
||||
transcodes.SetNodes([]*nodepool.Node{{
|
||||
ID: 2, Name: "transcode-1", Type: nodepool.NodeTypeTranscode, URL: deadNodeURL,
|
||||
Enabled: true, Healthy: true, Group: &group,
|
||||
}})
|
||||
handler.NodePlanner = nodepool.NewPlanner(proxies, transcodes)
|
||||
|
||||
sessionID := newRemoteAudioSwitchSession(t, sessionMgr, file, deadNodeURL)
|
||||
|
||||
changeReq := httptest.NewRequest(
|
||||
http.MethodPatch,
|
||||
"/api/v1/playback/"+sessionID+"/audio",
|
||||
strings.NewReader(`{"audio_track_index":1,"position":0}`),
|
||||
)
|
||||
changeReq = changeReq.WithContext(newAuthorizedPlaybackContext())
|
||||
changeReq = withPlaybackRouteParam(changeReq, "session_id", sessionID)
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleChangeAudioTrack(rr, changeReq)
|
||||
|
||||
// Chosen posture: surface the failure (mirror HandleStartTranscode) so a 200
|
||||
// never lies to the client about an audio switch that didn't happen.
|
||||
if rr.Code != http.StatusBadGateway {
|
||||
t.Fatalf("change status = %d, want %d, body = %s", rr.Code, http.StatusBadGateway, rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleStartTranscode_LocalPathPropagatesSelectedAudioTrack(t *testing.T) {
|
||||
sessionMgr := playback.NewSessionManager(0, 0)
|
||||
filePath := writePlaybackTestMediaFile(t, "movie.mkv")
|
||||
@@ -1213,7 +1452,7 @@ func TestHandleStartTranscode_LocalPathPropagatesSelectedAudioTrack(t *testing.T
|
||||
t.Fatalf("transcode status = %d, body = %s", transcodeRR.Code, transcodeRR.Body.String())
|
||||
}
|
||||
|
||||
transcodeSession := handler.getTranscodeSession(startResp.SessionID)
|
||||
transcodeSession := handler.tm.GetTranscodeSession(startResp.SessionID)
|
||||
if transcodeSession == nil {
|
||||
t.Fatal("expected local transcode session")
|
||||
}
|
||||
@@ -1264,7 +1503,7 @@ func TestHandleStartTranscode_MPEG2SeekedCopyRemainsCopyVideo(t *testing.T) {
|
||||
t.Fatalf("transcode status = %d, body = %s", transcodeRR.Code, transcodeRR.Body.String())
|
||||
}
|
||||
|
||||
transcodeSession := handler.getTranscodeSession(session.ID)
|
||||
transcodeSession := handler.tm.GetTranscodeSession(session.ID)
|
||||
if transcodeSession == nil {
|
||||
t.Fatal("expected local transcode session")
|
||||
}
|
||||
|
||||
@@ -503,7 +503,6 @@ func (h *PluginHandler) HandleCreateInstallation(w http.ResponseWriter, r *http.
|
||||
}
|
||||
|
||||
h.syncMetadataProviders(r.Context(), result.Installation)
|
||||
h.syncImageResolvers(r.Context(), result.Installation)
|
||||
|
||||
response, err := h.buildInstallationResponse(r.Context(), result.Installation, result.Manifest)
|
||||
if err != nil {
|
||||
@@ -662,7 +661,6 @@ func (h *PluginHandler) installUploadedPlugin(ctx context.Context, path string)
|
||||
|
||||
func (h *PluginHandler) writeUploadedPluginResponse(w http.ResponseWriter, r *http.Request, result *plugins.InstallResult) {
|
||||
h.syncMetadataProviders(r.Context(), result.Installation)
|
||||
h.syncImageResolvers(r.Context(), result.Installation)
|
||||
|
||||
response, err := h.buildInstallationResponse(r.Context(), result.Installation, result.Manifest)
|
||||
if err != nil {
|
||||
@@ -703,35 +701,6 @@ func (h *PluginHandler) syncMetadataProviders(ctx context.Context, installation
|
||||
}
|
||||
}
|
||||
|
||||
// syncImageResolvers registers image resolver sources for any metadata_provider.v1
|
||||
// capabilities from the given installation so that plugin-prefixed image URLs
|
||||
// (e.g. "tmdb://poster/abc.jpg") can be resolved at runtime.
|
||||
func (h *PluginHandler) syncImageResolvers(ctx context.Context, installation *plugins.Installation) {
|
||||
if h.imageResolver == nil || h.service == nil {
|
||||
return
|
||||
}
|
||||
|
||||
caps, err := h.installations.ListCapabilities(ctx, installation.ID)
|
||||
if err != nil {
|
||||
slog.Error("listing capabilities for image resolver sync",
|
||||
"installation_id", installation.ID, "error", err)
|
||||
return
|
||||
}
|
||||
|
||||
for _, cap := range caps {
|
||||
if cap.Type != "metadata_provider.v1" {
|
||||
continue
|
||||
}
|
||||
source := metadata.NewPluginClientSource(installation.ID, cap.ID, func(
|
||||
ctx context.Context, installationID int, capabilityID string,
|
||||
) (metadata.PluginMetadataClient, error) {
|
||||
return h.service.MetadataProviderClient(ctx, installationID, capabilityID)
|
||||
})
|
||||
h.imageResolver.RegisterSource(cap.ID, source)
|
||||
slog.Info("registered plugin image resolver", "capability_id", cap.ID, "installation_id", installation.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func isZipUpload(data []byte) bool {
|
||||
if len(data) < 4 {
|
||||
return false
|
||||
@@ -849,11 +818,6 @@ func (h *PluginHandler) HandleUpdateInstallation(w http.ResponseWriter, r *http.
|
||||
return
|
||||
}
|
||||
|
||||
// When a plugin is being enabled, register its image resolvers.
|
||||
if req.Enabled != nil && *req.Enabled && !currentInstallation.Enabled {
|
||||
h.syncImageResolvers(r.Context(), installation)
|
||||
}
|
||||
|
||||
response, err := h.buildInstallationResponse(r.Context(), installation, nil)
|
||||
if err != nil {
|
||||
slog.Error("building updated plugin installation response", "error", err)
|
||||
@@ -883,7 +847,6 @@ func (h *PluginHandler) HandleApplyUpdate(w http.ResponseWriter, r *http.Request
|
||||
}
|
||||
|
||||
h.syncMetadataProviders(r.Context(), installation)
|
||||
h.syncImageResolvers(r.Context(), installation)
|
||||
|
||||
response, err := h.buildInstallationResponse(r.Context(), installation, nil)
|
||||
if err != nil {
|
||||
|
||||
@@ -28,6 +28,12 @@ type ProfileHandler struct {
|
||||
ProfileTokens *access.ProfileTokenService
|
||||
AvatarStore profileAvatarStore
|
||||
AvatarTTL time.Duration
|
||||
// DeviceLibraryPurger removes a deleted profile's device rows (and, via
|
||||
// cascade, its managed downloads and subscriptions). Profiles may live
|
||||
// outside Postgres, so no FK cascade covers these shared tables.
|
||||
DeviceLibraryPurger interface {
|
||||
PurgeProfileDevices(ctx context.Context, userID int, profileID string) error
|
||||
}
|
||||
}
|
||||
|
||||
// NewProfileHandler creates a new ProfileHandler.
|
||||
@@ -589,6 +595,13 @@ func (h *ProfileHandler) HandleDeleteProfile(w http.ResponseWriter, r *http.Requ
|
||||
slog.Warn("profile avatar cleanup failed after delete", "user_id", userID, "profile_id", profileID, "error", cleanupErr)
|
||||
}
|
||||
}
|
||||
if h.DeviceLibraryPurger != nil {
|
||||
purgeCtx, cancel := context.WithTimeout(context.WithoutCancel(r.Context()), 30*time.Second)
|
||||
defer cancel()
|
||||
if purgeErr := h.DeviceLibraryPurger.PurgeProfileDevices(purgeCtx, userID, profileID); purgeErr != nil {
|
||||
slog.Warn("profile device-library purge failed after delete", "user_id", userID, "profile_id", profileID, "error", purgeErr)
|
||||
}
|
||||
}
|
||||
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
@@ -3,17 +3,51 @@ package handlers
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/access"
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
// progressClockSkew bounds how far ahead of server time a client-supplied
|
||||
// progress event time may sit before it is clamped to "now".
|
||||
const progressClockSkew = 2 * time.Minute
|
||||
|
||||
// parseClientEventTime parses an RFC3339 client event time. Malformed values
|
||||
// are an error the caller must reject: treating them as "now" would let a
|
||||
// stale offline event win LWW as a fresh server-time write.
|
||||
func parseClientEventTime(s string) (time.Time, error) {
|
||||
t, err := time.Parse(time.RFC3339, s)
|
||||
if err != nil {
|
||||
return time.Time{}, err
|
||||
}
|
||||
return t.UTC(), nil
|
||||
}
|
||||
|
||||
// clampEventAt bounds a client event time to at most now+skew: a value past the
|
||||
// window is clamped to now, so a skewed or malicious clock can at most claim
|
||||
// "now" for its own profile and never lock in a far-future LWW win (invariant 1).
|
||||
func clampEventAt(client, now time.Time) time.Time {
|
||||
if client.IsZero() {
|
||||
return now
|
||||
}
|
||||
if client.After(now.Add(progressClockSkew)) {
|
||||
return now
|
||||
}
|
||||
return client
|
||||
}
|
||||
|
||||
// ProgressLibraryLookup resolves which progress items belong to a library.
|
||||
type ProgressLibraryLookup interface {
|
||||
GetItemsInFolder(ctx context.Context, contentIDs []string, folderID int) (map[string]bool, error)
|
||||
// FilterAccessibleContentIDs returns the subset of contentIDs the viewer
|
||||
// may access given their library scope and content-rating ceiling.
|
||||
FilterAccessibleContentIDs(ctx context.Context, contentIDs []string, allowedFolderIDs, disabledFolderIDs []int, maxContentRating string) (map[string]bool, error)
|
||||
}
|
||||
|
||||
// ProgressHandler handles watch progress and sync endpoints.
|
||||
@@ -53,6 +87,8 @@ type progressEntryResponse struct {
|
||||
|
||||
type progressListResponse struct {
|
||||
Progress []progressEntryResponse `json:"progress"`
|
||||
// NextCursor is the opaque server token to resume a ?since= delta from.
|
||||
NextCursor string `json:"next_cursor,omitempty"`
|
||||
}
|
||||
|
||||
type syncProgressItem struct {
|
||||
@@ -60,6 +96,9 @@ type syncProgressItem struct {
|
||||
Position float64 `json:"position"`
|
||||
Duration float64 `json:"duration"`
|
||||
ForceOverwrite bool `json:"force_overwrite"`
|
||||
// UpdatedAt is the client EVENT time (RFC3339) for an offline-queued item.
|
||||
// The server clamps it to now+skew and uses it only as the LWW key.
|
||||
UpdatedAt *string `json:"updated_at,omitempty"`
|
||||
}
|
||||
|
||||
type syncProgressRequest struct {
|
||||
@@ -90,6 +129,7 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
status := r.URL.Query().Get("status")
|
||||
since := r.URL.Query().Get("since")
|
||||
limit, offset := parsePagination(r)
|
||||
libraryID, err := parseLibraryIDParam(r)
|
||||
if err != nil {
|
||||
@@ -97,11 +137,39 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
return
|
||||
}
|
||||
|
||||
entries, err := store.ListProgress(r.Context(), profileID, status, limit, offset)
|
||||
// A ?since= cursor switches to server-ordered delta delivery (rows changed
|
||||
// elsewhere since the cursor), immune to client clock skew. Absent since →
|
||||
// today's status/pagination listing.
|
||||
var entries []userstore.WatchProgress
|
||||
var nextCursor string
|
||||
if since != "" {
|
||||
entries, nextCursor, err = store.ListProgressSince(r.Context(), profileID, since)
|
||||
} else {
|
||||
entries, err = store.ListProgress(r.Context(), profileID, status, limit, offset)
|
||||
}
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list progress")
|
||||
return
|
||||
}
|
||||
|
||||
// Drop entries the viewer can't access before they reach the client.
|
||||
// Without this, a library-restricted profile receives progress rows for
|
||||
// items outside its scope (e.g. an XXX title) and the client then fans out
|
||||
// per-item detail fetches that 404 — a dead Continue Watching tile. Only
|
||||
// runs for restricted profiles; unrestricted viewers are unaffected.
|
||||
if scope, ok := access.GetScope(r.Context()); ok &&
|
||||
(scope.AllowedLibraryIDs != nil || len(scope.DisabledLibraryIDs) > 0 || scope.MaxContentRating != "") {
|
||||
if h.LibraryLookup == nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply access filter")
|
||||
return
|
||||
}
|
||||
entries, err = filterProgressEntriesByAccess(r.Context(), entries, scope, h.LibraryLookup)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply access filter")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
if libraryID > 0 {
|
||||
if h.LibraryLookup == nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to apply library filter")
|
||||
@@ -115,7 +183,8 @@ func (h *ProgressHandler) HandleListProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
resp := progressListResponse{
|
||||
Progress: make([]progressEntryResponse, 0, len(entries)),
|
||||
Progress: make([]progressEntryResponse, 0, len(entries)),
|
||||
NextCursor: nextCursor,
|
||||
}
|
||||
for _, e := range entries {
|
||||
resp.Progress = append(resp.Progress, progressEntryResponse{
|
||||
@@ -144,6 +213,27 @@ func parseLibraryIDParam(r *http.Request) (int, error) {
|
||||
return libraryID, nil
|
||||
}
|
||||
|
||||
// progressContentIDs collects the media item IDs from a progress slice.
|
||||
func progressContentIDs(entries []userstore.WatchProgress) []string {
|
||||
contentIDs := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
contentIDs = append(contentIDs, entry.MediaItemID)
|
||||
}
|
||||
return contentIDs
|
||||
}
|
||||
|
||||
// keepAccessibleEntries returns, in order, the entries whose media item ID maps
|
||||
// to true in accessible.
|
||||
func keepAccessibleEntries(entries []userstore.WatchProgress, accessible map[string]bool) []userstore.WatchProgress {
|
||||
filtered := make([]userstore.WatchProgress, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if accessible[entry.MediaItemID] {
|
||||
filtered = append(filtered, entry)
|
||||
}
|
||||
}
|
||||
return filtered
|
||||
}
|
||||
|
||||
func filterProgressEntriesByLibrary(
|
||||
ctx context.Context,
|
||||
entries []userstore.WatchProgress,
|
||||
@@ -154,24 +244,33 @@ func filterProgressEntriesByLibrary(
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
contentIDs := make([]string, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
contentIDs = append(contentIDs, entry.MediaItemID)
|
||||
}
|
||||
|
||||
allowed, err := lookup.GetItemsInFolder(ctx, contentIDs, libraryID)
|
||||
allowed, err := lookup.GetItemsInFolder(ctx, progressContentIDs(entries), libraryID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
filtered := make([]userstore.WatchProgress, 0, len(entries))
|
||||
for _, entry := range entries {
|
||||
if allowed[entry.MediaItemID] {
|
||||
filtered = append(filtered, entry)
|
||||
}
|
||||
return keepAccessibleEntries(entries, allowed), nil
|
||||
}
|
||||
|
||||
// filterProgressEntriesByAccess removes progress entries whose item falls
|
||||
// outside the viewer's access scope (allowed/disabled libraries and the
|
||||
// content-rating ceiling).
|
||||
func filterProgressEntriesByAccess(
|
||||
ctx context.Context,
|
||||
entries []userstore.WatchProgress,
|
||||
scope access.Scope,
|
||||
lookup ProgressLibraryLookup,
|
||||
) ([]userstore.WatchProgress, error) {
|
||||
if len(entries) == 0 {
|
||||
return entries, nil
|
||||
}
|
||||
|
||||
return filtered, nil
|
||||
accessible, err := lookup.FilterAccessibleContentIDs(ctx, progressContentIDs(entries), scope.AllowedLibraryIDs, scope.DisabledLibraryIDs, scope.MaxContentRating)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return keepAccessibleEntries(entries, accessible), nil
|
||||
}
|
||||
|
||||
// HandleSyncProgress handles POST /sync/progress.
|
||||
@@ -227,9 +326,32 @@ func (h *ProgressHandler) HandleSyncProgress(w http.ResponseWriter, r *http.Requ
|
||||
}
|
||||
|
||||
var updateErr error
|
||||
if item.ForceOverwrite {
|
||||
switch {
|
||||
case item.UpdatedAt != nil:
|
||||
// Offline-queued event: clamp the client event time and merge
|
||||
// last-write-wins on the bounded event_at. synced_seq (the cursor) is
|
||||
// stamped server-side; completion still comes from the threshold logic,
|
||||
// never the timestamp alone.
|
||||
client, parseErr := parseClientEventTime(*item.UpdatedAt)
|
||||
if parseErr != nil {
|
||||
result.Status = "error"
|
||||
result.Error = "updated_at must be RFC3339"
|
||||
results = append(results, result)
|
||||
continue
|
||||
}
|
||||
now := time.Now()
|
||||
eventAt := clampEventAt(client, now)
|
||||
if !client.IsZero() && client.After(now.Add(progressClockSkew)) {
|
||||
slog.Warn("clamped future-dated progress event time",
|
||||
"profile_id", profileID, "media_item_id", item.MediaItemID)
|
||||
}
|
||||
pos, completed, skip := userstore.ResolveProgressState(item.Position, item.Duration, thresholds)
|
||||
if !skip {
|
||||
_, updateErr = store.SetProgressIfNewer(r.Context(), profileID, item.MediaItemID, pos, item.Duration, completed, eventAt)
|
||||
}
|
||||
case item.ForceOverwrite:
|
||||
updateErr = store.SetProgress(r.Context(), profileID, item.MediaItemID, item.Position, item.Duration, thresholds)
|
||||
} else {
|
||||
default:
|
||||
updateErr = store.UpdateProgress(r.Context(), profileID, item.MediaItemID, item.Position, item.Duration, thresholds)
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/access"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
// fakeProgressLookup records the arguments it was called with and returns a
|
||||
// fixed accessibility map.
|
||||
type fakeProgressLookup struct {
|
||||
accessible map[string]bool
|
||||
|
||||
gotContentIDs []string
|
||||
gotAllowed []int
|
||||
gotDisabled []int
|
||||
gotRating string
|
||||
}
|
||||
|
||||
func (f *fakeProgressLookup) GetItemsInFolder(context.Context, []string, int) (map[string]bool, error) {
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
func (f *fakeProgressLookup) FilterAccessibleContentIDs(
|
||||
_ context.Context, contentIDs []string, allowedFolderIDs, disabledFolderIDs []int, maxContentRating string,
|
||||
) (map[string]bool, error) {
|
||||
f.gotContentIDs = contentIDs
|
||||
f.gotAllowed = allowedFolderIDs
|
||||
f.gotDisabled = disabledFolderIDs
|
||||
f.gotRating = maxContentRating
|
||||
return f.accessible, nil
|
||||
}
|
||||
|
||||
func entries(ids ...string) []userstore.WatchProgress {
|
||||
out := make([]userstore.WatchProgress, 0, len(ids))
|
||||
for _, id := range ids {
|
||||
out = append(out, userstore.WatchProgress{MediaItemID: id})
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func ids(entries []userstore.WatchProgress) []string {
|
||||
out := make([]string, 0, len(entries))
|
||||
for _, e := range entries {
|
||||
out = append(out, e.MediaItemID)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func TestFilterProgressEntriesByAccess(t *testing.T) {
|
||||
// access.Scope sets DisabledLibraryIDs only when AllowedLibraryIDs is nil
|
||||
// (see access.Scope docs), so the two restriction shapes are tested
|
||||
// separately rather than as one impossible combined scope.
|
||||
cases := []struct {
|
||||
name string
|
||||
scope access.Scope
|
||||
}{
|
||||
{
|
||||
name: "allowed libraries + rating",
|
||||
scope: access.Scope{AllowedLibraryIDs: []int{1, 2}, MaxContentRating: "PG-13"},
|
||||
},
|
||||
{
|
||||
name: "disabled libraries + rating",
|
||||
scope: access.Scope{DisabledLibraryIDs: []int{9}, MaxContentRating: "PG-13"},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
lookup := &fakeProgressLookup{accessible: map[string]bool{"a": true, "c": true}}
|
||||
|
||||
got, err := filterProgressEntriesByAccess(context.Background(), entries("a", "b", "c"), tc.scope, lookup)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
|
||||
want := []string{"a", "c"}
|
||||
if g := ids(got); len(g) != len(want) || g[0] != want[0] || g[1] != want[1] {
|
||||
t.Fatalf("filtered entries = %v, want %v", g, want)
|
||||
}
|
||||
|
||||
// Scope must be forwarded verbatim to the lookup.
|
||||
if !slices.Equal(lookup.gotAllowed, tc.scope.AllowedLibraryIDs) {
|
||||
t.Errorf("allowed folders = %v, want %v", lookup.gotAllowed, tc.scope.AllowedLibraryIDs)
|
||||
}
|
||||
if !slices.Equal(lookup.gotDisabled, tc.scope.DisabledLibraryIDs) {
|
||||
t.Errorf("disabled folders = %v, want %v", lookup.gotDisabled, tc.scope.DisabledLibraryIDs)
|
||||
}
|
||||
if len(lookup.gotContentIDs) != 3 {
|
||||
t.Errorf("content ids = %v, want 3 entries", lookup.gotContentIDs)
|
||||
}
|
||||
if lookup.gotRating != tc.scope.MaxContentRating {
|
||||
t.Errorf("max content rating = %q, want %q", lookup.gotRating, tc.scope.MaxContentRating)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterProgressEntriesByAccessEmpty(t *testing.T) {
|
||||
lookup := &fakeProgressLookup{accessible: map[string]bool{}}
|
||||
got, err := filterProgressEntriesByAccess(context.Background(), nil, access.Scope{}, lookup)
|
||||
if err != nil {
|
||||
t.Fatalf("unexpected error: %v", err)
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("expected no entries, got %v", ids(got))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
func TestClampEventAt(t *testing.T) {
|
||||
now := time.Date(2026, 6, 19, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
client time.Time
|
||||
want time.Time
|
||||
}{
|
||||
{"zero defaults to now", time.Time{}, now},
|
||||
{"far future clamps to now", now.Add(48 * time.Hour), now},
|
||||
{"just past skew clamps to now", now.Add(progressClockSkew + time.Minute), now},
|
||||
{"within skew is kept", now.Add(time.Minute), now.Add(time.Minute)},
|
||||
{"past is kept", now.Add(-3 * time.Hour), now.Add(-3 * time.Hour)},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
if got := clampEventAt(tc.client, now); !got.Equal(tc.want) {
|
||||
t.Fatalf("clampEventAt(%v) = %v, want %v", tc.client, got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestOfflineProgressState(t *testing.T) {
|
||||
thresholds := userstore.ProgressThresholds{WatchedPct: 90, MinResumePct: 5}
|
||||
|
||||
// Below the min-resume floor → skipped.
|
||||
if _, _, skip := userstore.ResolveProgressState(10, 1000, thresholds); !skip {
|
||||
t.Fatal("tiny progress should be skipped")
|
||||
}
|
||||
// Above the watched threshold → completed latch, position reset.
|
||||
pos, completed, skip := userstore.ResolveProgressState(950, 1000, thresholds)
|
||||
if skip || !completed || pos != 0 {
|
||||
t.Fatalf("watched item = (pos=%v completed=%v skip=%v), want (0 true false)", pos, completed, skip)
|
||||
}
|
||||
// Normal mid-progress → kept as-is.
|
||||
pos, completed, skip = userstore.ResolveProgressState(500, 1000, thresholds)
|
||||
if skip || completed || pos != 500 {
|
||||
t.Fatalf("mid progress = (pos=%v completed=%v skip=%v), want (500 false false)", pos, completed, skip)
|
||||
}
|
||||
}
|
||||
@@ -764,6 +764,8 @@ func (h *SectionHandler) loadResolvedHomeSections(r *http.Request) ([]sections.R
|
||||
}
|
||||
}
|
||||
|
||||
resolved = filterResolvedSectionsByAccess(resolved, accessFilter)
|
||||
|
||||
return resolved, libraryIDs, accessFilter, profileID, nil
|
||||
}
|
||||
|
||||
@@ -1047,6 +1049,7 @@ func (h *SectionHandler) HandleSectionSettings(w http.ResponseWriter, r *http.Re
|
||||
}
|
||||
|
||||
resolved := sections.ResolveForSettings(adminSections, overrides)
|
||||
resolved = filterResolvedSectionsByAccess(resolved, requestAccessFilter(r))
|
||||
|
||||
type settingsEntry struct {
|
||||
ID string `json:"id"`
|
||||
@@ -1080,6 +1083,64 @@ func (h *SectionHandler) HandleSectionSettings(w http.ResponseWriter, r *http.Re
|
||||
writeJSON(w, http.StatusOK, map[string][]settingsEntry{"sections": entries})
|
||||
}
|
||||
|
||||
func filterResolvedSectionsByAccess(resolved []sections.ResolvedSection, filter catalog.AccessFilter) []sections.ResolvedSection {
|
||||
if filter.AllowedLibraryIDs == nil && len(filter.DisabledLibraryIDs) == 0 {
|
||||
return resolved
|
||||
}
|
||||
|
||||
out := resolved[:0]
|
||||
for _, section := range resolved {
|
||||
if sectionAllowedByAccess(section, filter) {
|
||||
out = append(out, section)
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func sectionAllowedByAccess(section sections.ResolvedSection, filter catalog.AccessFilter) bool {
|
||||
configLibraryIDs := sections.ParseConfigFilters(section.Config).LibraryIDs()
|
||||
if len(configLibraryIDs) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
if filter.AllowedLibraryIDs != nil {
|
||||
return intSlicesIntersect(configLibraryIDs, filter.AllowedLibraryIDs)
|
||||
}
|
||||
|
||||
for _, libraryID := range configLibraryIDs {
|
||||
if !intSliceContains(filter.DisabledLibraryIDs, libraryID) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func intSlicesIntersect(left, right []int) bool {
|
||||
if len(left) == 0 || len(right) == 0 {
|
||||
return false
|
||||
}
|
||||
|
||||
set := make(map[int]struct{}, len(right))
|
||||
for _, value := range right {
|
||||
set[value] = struct{}{}
|
||||
}
|
||||
for _, value := range left {
|
||||
if _, ok := set[value]; ok {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func intSliceContains(values []int, target int) bool {
|
||||
for _, value := range values {
|
||||
if value == target {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// applyDiversityFilter removes items from sections whose recipe has
|
||||
// AvoidDuplicates=true if the same content ID was already surfaced by an
|
||||
// earlier section in the same render. Operates in-place — preserves order.
|
||||
@@ -1179,6 +1240,7 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
|
||||
userStates := h.listSectionItemUserStates(r, allItems)
|
||||
imageURLs := h.resolveSectionItemImageURLs(r.Context(), withItems)
|
||||
episodeMeta := h.listSectionEpisodeItemMeta(r.Context(), withItems, requestAccessFilter(r))
|
||||
mangaChapterMeta := h.listSectionMangaChapterItemMeta(r.Context(), allItems)
|
||||
for _, s := range withItems {
|
||||
items := make([]sectionItemResponse, 0, len(s.Items))
|
||||
for _, item := range s.Items {
|
||||
@@ -1193,6 +1255,17 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
|
||||
meta = &value
|
||||
}
|
||||
}
|
||||
// Manga chapters carry their series linkage on top of whatever
|
||||
// meta (e.g. reading progress) the section already resolved, so
|
||||
// continue-reading cards can head to the series.
|
||||
if value, ok := mangaChapterMeta[item.ContentID]; ok {
|
||||
if meta == nil {
|
||||
empty := sections.SectionItemMeta{}
|
||||
meta = &empty
|
||||
}
|
||||
meta.SeriesID = value.SeriesID
|
||||
meta.SeriesTitle = value.SeriesTitle
|
||||
}
|
||||
imageKey := sectionItemImageKey{sectionID: s.ID, contentID: item.ContentID}
|
||||
items = append(items, h.toSectionItemResponse(s.SectionType, item, meta, overlaySummaries[item.ContentID], userStates[item.ContentID], imageURLs[imageKey]))
|
||||
}
|
||||
@@ -1211,6 +1284,33 @@ func (h *SectionHandler) buildSectionsResponse(r *http.Request, withItems []sect
|
||||
return resp
|
||||
}
|
||||
|
||||
// listSectionMangaChapterItemMeta resolves series linkage for every manga
|
||||
// chapter (type='ebook' linked via manga_chapters) among the section items.
|
||||
// Non-chapter ebooks simply get no entry.
|
||||
func (h *SectionHandler) listSectionMangaChapterItemMeta(ctx context.Context, items []*models.MediaItem) map[string]sections.SectionItemMeta {
|
||||
if h == nil || h.fetcher == nil {
|
||||
return map[string]sections.SectionItemMeta{}
|
||||
}
|
||||
ids := make([]string, 0)
|
||||
seen := make(map[string]struct{})
|
||||
for _, item := range items {
|
||||
if item == nil || item.Type != "ebook" || strings.TrimSpace(item.ContentID) == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[item.ContentID]; ok {
|
||||
continue
|
||||
}
|
||||
seen[item.ContentID] = struct{}{}
|
||||
ids = append(ids, item.ContentID)
|
||||
}
|
||||
meta, err := h.fetcher.FetchMangaChapterSeriesMeta(ctx, ids)
|
||||
if err != nil {
|
||||
slog.Warn("loading section manga chapter metadata", "error", err)
|
||||
return map[string]sections.SectionItemMeta{}
|
||||
}
|
||||
return meta
|
||||
}
|
||||
|
||||
func (h *SectionHandler) listSectionEpisodeItemMeta(ctx context.Context, withItems []sections.SectionWithItems, filter catalog.AccessFilter) map[string]sections.SectionItemMeta {
|
||||
if h == nil || h.episodeFetcher == nil {
|
||||
return map[string]sections.SectionItemMeta{}
|
||||
|
||||
@@ -390,6 +390,68 @@ func TestLibraryDefaultSectionsUsesFolderType(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterResolvedSectionsByAccessHidesBlockedLibraryRows(t *testing.T) {
|
||||
sectionsIn := []sections.ResolvedSection{
|
||||
{
|
||||
ID: "continue",
|
||||
SectionType: sections.SectionContinueWatching,
|
||||
Title: "Continue Watching",
|
||||
Config: sections.ContinueTypeConfig(sections.ContinueTypeWatching),
|
||||
},
|
||||
{
|
||||
ID: "allowed-library",
|
||||
SectionType: sections.SectionRecentlyAdded,
|
||||
Title: "Recently Added in Allowed",
|
||||
Config: sections.GeneratedHomeLibraryRecentConfig(11),
|
||||
},
|
||||
{
|
||||
ID: "blocked-library",
|
||||
SectionType: sections.SectionRecentlyAdded,
|
||||
Title: "Recently Added in Blocked",
|
||||
Config: sections.GeneratedHomeLibraryRecentConfig(42),
|
||||
},
|
||||
}
|
||||
|
||||
got := filterResolvedSectionsByAccess(sectionsIn, catalog.AccessFilter{
|
||||
AllowedLibraryIDs: []int{11},
|
||||
})
|
||||
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("filtered sections length = %d, want 2", len(got))
|
||||
}
|
||||
if got[0].ID != "continue" || got[1].ID != "allowed-library" {
|
||||
t.Fatalf("filtered section ids = [%s %s], want [continue allowed-library]", got[0].ID, got[1].ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFilterResolvedSectionsByAccessHidesDisabledOnlyRows(t *testing.T) {
|
||||
sectionsIn := []sections.ResolvedSection{
|
||||
{
|
||||
ID: "disabled-library",
|
||||
SectionType: sections.SectionRecentlyReleased,
|
||||
Title: "Recently Released in Disabled",
|
||||
Config: sections.GeneratedHomeLibraryRecentConfig(42),
|
||||
},
|
||||
{
|
||||
ID: "mixed-libraries",
|
||||
SectionType: sections.SectionRecentlyReleased,
|
||||
Title: "Mixed",
|
||||
Config: []byte(`{"filter_library_ids":[11,42]}`),
|
||||
},
|
||||
}
|
||||
|
||||
got := filterResolvedSectionsByAccess(sectionsIn, catalog.AccessFilter{
|
||||
DisabledLibraryIDs: []int{42},
|
||||
})
|
||||
|
||||
if len(got) != 1 {
|
||||
t.Fatalf("filtered sections length = %d, want 1", len(got))
|
||||
}
|
||||
if got[0].ID != "mixed-libraries" {
|
||||
t.Fatalf("filtered section id = %s, want mixed-libraries", got[0].ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyDiversityFilterSkipsDuplicatesInLaterAvoidSection(t *testing.T) {
|
||||
in := []sections.SectionWithItems{
|
||||
{ResolvedSection: sections.ResolvedSection{ID: "ra", SectionType: sections.SectionRecentlyAdded}, Items: []*models.MediaItem{{ContentID: "abc"}}},
|
||||
|
||||
@@ -182,7 +182,7 @@ func (h *PlaybackHandler) handleRealtimeClientMessage(sessionID string, data []b
|
||||
}
|
||||
switch record.Name {
|
||||
case playback.CommandStop, playback.CommandTerminate:
|
||||
err := h.stopPlaybackSessionByID(context.Background(), sessionID)
|
||||
err := h.stopPlaybackSessionByID(context.Background(), sessionID, true)
|
||||
if err != nil && !errors.Is(err, playback.ErrSessionNotFound) {
|
||||
slog.Error("failed to stop playback after realtime completion", "session", sessionID, "playback_session_id", sessionID, "error", err)
|
||||
}
|
||||
|
||||
@@ -9,13 +9,23 @@ import (
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/cache"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
// deviceSeenThrottle bounds how often a device's last_seen_at is refreshed from
|
||||
// request traffic. Device-setting reads each registered the device (an upsert
|
||||
// on a single per-device row), so a page that fetches many settings serialized
|
||||
// hundreds of upserts on that row's lock. Skipping the upsert when the device
|
||||
// was seen within this window removes the contention while keeping last_seen
|
||||
// fresh to within the window.
|
||||
const deviceSeenThrottle = 5 * time.Minute
|
||||
|
||||
const subtitleAppearanceSettingKey = "subtitle_appearance"
|
||||
const (
|
||||
libraryPageStateSettingKey = "ui.library_page_state"
|
||||
@@ -38,11 +48,32 @@ type ServerSettingReader interface {
|
||||
type SettingsHandler struct {
|
||||
storeProvider userstore.UserStoreProvider
|
||||
serverSettings ServerSettingReader
|
||||
deviceSeen *cache.TTLCache[struct{}]
|
||||
}
|
||||
|
||||
// NewSettingsHandler creates a new SettingsHandler.
|
||||
func NewSettingsHandler(provider userstore.UserStoreProvider) *SettingsHandler {
|
||||
return &SettingsHandler{storeProvider: provider}
|
||||
return &SettingsHandler{
|
||||
storeProvider: provider,
|
||||
deviceSeen: cache.NewTTLCache[struct{}](),
|
||||
}
|
||||
}
|
||||
|
||||
// shouldRegisterDevice reports whether the device's last_seen_at should be
|
||||
// refreshed now, throttling to one upsert per deviceSeenThrottle window per
|
||||
// (profile, device). It marks the device seen before returning true so a burst
|
||||
// of concurrent reads collapses to a single upsert instead of contending on the
|
||||
// device row.
|
||||
func (h *SettingsHandler) shouldRegisterDevice(profileID, deviceID string) bool {
|
||||
if h == nil || h.deviceSeen == nil {
|
||||
return true
|
||||
}
|
||||
key := profileID + "\x00" + deviceID
|
||||
if _, seen := h.deviceSeen.Get(key); seen {
|
||||
return false
|
||||
}
|
||||
h.deviceSeen.Set(key, struct{}{}, deviceSeenThrottle)
|
||||
return true
|
||||
}
|
||||
|
||||
// SetServerSettings configures the optional server settings reader for overlay config etc.
|
||||
@@ -382,7 +413,7 @@ func (h *SettingsHandler) HandleGetDeviceSetting(w http.ResponseWriter, r *http.
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
|
||||
return
|
||||
}
|
||||
registerRequestDevice(r.Context(), store, profileID, device)
|
||||
h.registerRequestDevice(r.Context(), store, profileID, device)
|
||||
|
||||
value, err := store.GetDeviceSetting(r.Context(), profileID, device.DeviceID, key)
|
||||
if err != nil {
|
||||
@@ -482,7 +513,7 @@ func (h *SettingsHandler) HandleDeleteDeviceSetting(w http.ResponseWriter, r *ht
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
|
||||
return
|
||||
}
|
||||
registerRequestDevice(r.Context(), store, profileID, device)
|
||||
h.registerRequestDevice(r.Context(), store, profileID, device)
|
||||
|
||||
if err := store.DeleteDeviceSetting(r.Context(), profileID, device.DeviceID, key); err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to delete device setting")
|
||||
@@ -512,7 +543,7 @@ func (h *SettingsHandler) HandleGetEffectiveSettings(w http.ResponseWriter, r *h
|
||||
}
|
||||
|
||||
device := deviceMetadataFromRequest(r)
|
||||
registerRequestDevice(r.Context(), store, profileID, device)
|
||||
h.registerRequestDevice(r.Context(), store, profileID, device)
|
||||
keys := parseSettingKeys(keysParam)
|
||||
resp := effectiveSettingsResponse{
|
||||
Settings: make([]effectiveSettingResponse, 0, len(keys)),
|
||||
@@ -543,7 +574,7 @@ func (h *SettingsHandler) HandleGetEffectiveSubtitleAppearance(w http.ResponseWr
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to access user store")
|
||||
return
|
||||
}
|
||||
registerRequestDevice(r.Context(), store, profileID, device)
|
||||
h.registerRequestDevice(r.Context(), store, profileID, device)
|
||||
|
||||
resolved, err := h.resolveEffectiveSetting(r.Context(), store, profileID, device, subtitleAppearanceSettingKey)
|
||||
if err != nil {
|
||||
@@ -604,7 +635,7 @@ func deviceMetadataFromRequest(r *http.Request) requestDeviceMetadata {
|
||||
}
|
||||
}
|
||||
|
||||
func registerRequestDevice(
|
||||
func (h *SettingsHandler) registerRequestDevice(
|
||||
ctx context.Context,
|
||||
store userstore.UserStore,
|
||||
profileID string,
|
||||
@@ -616,6 +647,9 @@ func registerRequestDevice(
|
||||
if store == nil {
|
||||
return
|
||||
}
|
||||
if !h.shouldRegisterDevice(profileID, device.DeviceID) {
|
||||
return
|
||||
}
|
||||
registry, ok := store.(userstore.DeviceRegistry)
|
||||
if !ok {
|
||||
return
|
||||
|
||||
@@ -48,7 +48,8 @@ func (r testAdminUserRepo) GetByID(_ context.Context, id int) (*models.User, err
|
||||
}
|
||||
|
||||
func TestRegisterRequestDeviceNilStore(t *testing.T) {
|
||||
registerRequestDevice(context.Background(), nil, "profile-1", requestDeviceMetadata{
|
||||
h := NewSettingsHandler(nil)
|
||||
h.registerRequestDevice(context.Background(), nil, "profile-1", requestDeviceMetadata{
|
||||
DeviceID: "device-1",
|
||||
DeviceName: "Living Room",
|
||||
DevicePlatform: "web",
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/cache"
|
||||
)
|
||||
|
||||
// Device-setting reads register the device (last_seen_at upsert). A page that
|
||||
// fetches many settings must not issue one upsert per read for the same device
|
||||
// — they contend on a single row. The throttle collapses repeats within the
|
||||
// window to one registration.
|
||||
func TestShouldRegisterDevice_ThrottlesRepeatWithinWindow(t *testing.T) {
|
||||
h := &SettingsHandler{deviceSeen: cache.NewTTLCache[struct{}]()}
|
||||
|
||||
if !h.shouldRegisterDevice("p1", "dev1") {
|
||||
t.Fatal("first sighting of a device should register")
|
||||
}
|
||||
if h.shouldRegisterDevice("p1", "dev1") {
|
||||
t.Fatal("repeat sighting within the window should be throttled (no upsert)")
|
||||
}
|
||||
if !h.shouldRegisterDevice("p1", "dev2") {
|
||||
t.Fatal("a different device on the same profile should register")
|
||||
}
|
||||
if !h.shouldRegisterDevice("p2", "dev1") {
|
||||
t.Fatal("the same device id under a different profile should register")
|
||||
}
|
||||
}
|
||||
@@ -32,6 +32,15 @@ type StreamHandler struct {
|
||||
EventsHub *evt.Hub
|
||||
AdminStore PlaybackAdminStore
|
||||
SessionSyncer PlaybackSessionSyncer
|
||||
// TM is the shared transcode/reconstruct manager (same instance as the
|
||||
// PlaybackHandler's). It lets a direct/remux stream rebuild its playback
|
||||
// Session from the recipe card after a server restart instead of 404-ing.
|
||||
// May be nil (tests / minimal setups) — reconstruct is then simply off.
|
||||
TM *playback.TranscodeManager
|
||||
// JWTSecret verifies the stream token carried on the serve URL (?st=), which
|
||||
// is the reconstruction descriptor for direct/remux after a restart. Empty
|
||||
// disables token-based reconstruct (tests / minimal setups).
|
||||
JWTSecret string
|
||||
// PlaybackConfig returns the current playback config; read it through
|
||||
// ffmpegPath(). May be nil (tests).
|
||||
PlaybackConfig func() config.PlaybackConfig
|
||||
@@ -54,6 +63,10 @@ func NewStreamHandler(sessionMgr SessionManagerInterface, fileResolver FilePathR
|
||||
return &StreamHandler{
|
||||
sessionMgr: sessionMgr,
|
||||
fileResolver: fileResolver,
|
||||
// A bare manager (no recipe store) behaves as "no reconstruct" — plain
|
||||
// GetSession + ownership — so HandleStream has a single code path. The
|
||||
// router overwrites this with the shared manager to enable reconstruct.
|
||||
TM: playback.NewTranscodeManager(),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -75,14 +88,22 @@ func (h *StreamHandler) HandleStream(w http.ResponseWriter, r *http.Request) {
|
||||
}
|
||||
setPlaybackSessionLogContext(r, sessionID)
|
||||
|
||||
session, err := h.sessionMgr.GetSession(sessionID)
|
||||
if err != nil {
|
||||
// Look up the session, reconstructing it from the recipe card on a not-found
|
||||
// miss (e.g. after a server restart) so a direct/remux stream resumes instead
|
||||
// of 404-ing. The client re-supplies its position (HTTP Range for direct, the
|
||||
// ?seek= query for remux), so no runtime beyond the Session needs rebuilding.
|
||||
// Without a token (or signing secret) reconstruct is off, collapsing to a
|
||||
// plain GetSession + ownership check.
|
||||
card := streamCardFromToken(r.URL.Query().Get(streamTokenParam), sessionID, h.JWTSecret)
|
||||
session, status := h.TM.LoadOrReconstructSession(r.Context(), h.sessionMgr.GetSession, sessionID, userID, card)
|
||||
switch status {
|
||||
case playback.SessionMissing:
|
||||
writePlaybackSessionNotFound(w)
|
||||
return
|
||||
}
|
||||
|
||||
// Verify session ownership.
|
||||
if session.UserID != userID {
|
||||
case playback.SessionLoadFailed:
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to load playback session")
|
||||
return
|
||||
case playback.SessionForbidden:
|
||||
writeError(w, http.StatusForbidden, "forbidden", "Session belongs to another user")
|
||||
return
|
||||
}
|
||||
@@ -240,37 +261,49 @@ func (h *StreamHandler) HandleSubtitle(w http.ResponseWriter, r *http.Request) {
|
||||
// Check downloaded subtitles (from S3).
|
||||
if h.SubtitleRepo != nil && h.S3Client != nil {
|
||||
downloaded, err := h.SubtitleRepo.ListDownloadedSubtitles(r.Context(), file.ID)
|
||||
if err == nil {
|
||||
downloadedIndex := embeddedIndex - len(file.SubtitleTracks)
|
||||
if downloadedIndex >= 0 && downloadedIndex < len(downloaded) {
|
||||
dl := downloaded[downloadedIndex]
|
||||
data, err := h.S3Client.GetObject(r.Context(), h.S3Bucket, dl.S3Key)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "s3_error", "Failed to load subtitle from storage")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
// A DB failure here must not masquerade as "track not found":
|
||||
// surface it as an internal error (with a server-side signal)
|
||||
// so the real failure is diagnosable instead of looking like an
|
||||
// intermittent 404 to the client.
|
||||
slog.ErrorContext(r.Context(), "list downloaded subtitles failed",
|
||||
"file_id", file.ID,
|
||||
"track", trackIndex,
|
||||
"error", err,
|
||||
)
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to list downloaded subtitles")
|
||||
return
|
||||
}
|
||||
|
||||
// Serve ASS/SSA downloaded subtitles as raw data.
|
||||
if playback.IsASS(string(dl.Format)) {
|
||||
playback.ServeSubtitle(w, data, "ass")
|
||||
return
|
||||
}
|
||||
|
||||
// If the subtitle is already VTT, serve directly.
|
||||
if dl.Format == subtitles.FormatVTT {
|
||||
playback.ServeSubtitle(w, data, "vtt")
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to VTT using the playback conversion pipeline.
|
||||
vttData, err := playback.ConvertToVTT(data, string(dl.Format))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "convert_error", "Failed to convert subtitle")
|
||||
return
|
||||
}
|
||||
playback.ServeSubtitle(w, vttData, "vtt")
|
||||
downloadedIndex := embeddedIndex - len(file.SubtitleTracks)
|
||||
if downloadedIndex >= 0 && downloadedIndex < len(downloaded) {
|
||||
dl := downloaded[downloadedIndex]
|
||||
data, err := h.S3Client.GetObject(r.Context(), h.S3Bucket, dl.S3Key)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadGateway, "s3_error", "Failed to load subtitle from storage")
|
||||
return
|
||||
}
|
||||
|
||||
// Serve ASS/SSA downloaded subtitles as raw data.
|
||||
if playback.IsASS(string(dl.Format)) {
|
||||
playback.ServeSubtitle(w, data, "ass")
|
||||
return
|
||||
}
|
||||
|
||||
// If the subtitle is already VTT, serve directly.
|
||||
if dl.Format == subtitles.FormatVTT {
|
||||
playback.ServeSubtitle(w, data, "vtt")
|
||||
return
|
||||
}
|
||||
|
||||
// Convert to VTT using the playback conversion pipeline.
|
||||
vttData, err := playback.ConvertToVTT(data, string(dl.Format))
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "convert_error", "Failed to convert subtitle")
|
||||
return
|
||||
}
|
||||
playback.ServeSubtitle(w, vttData, "vtt")
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,12 +2,15 @@ package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/playback"
|
||||
)
|
||||
@@ -121,6 +124,55 @@ func TestHandleStream_KeepsSessionWhenLookupFailsForNonMissingReason(t *testing.
|
||||
}
|
||||
}
|
||||
|
||||
// TestHandleSubtitle_ListDownloadedSubtitlesErrorReturns500 pins the fix for
|
||||
// issue #248: a failure listing downloaded subtitles must surface as a 500 with
|
||||
// an "internal_error" code, not be swallowed and reported to the client as a
|
||||
// generic "Subtitle track not found" 404 (which made a real backing-store
|
||||
// failure look like an intermittent client-side subtitle bug).
|
||||
func TestHandleSubtitle_ListDownloadedSubtitlesErrorReturns500(t *testing.T) {
|
||||
// No external or embedded tracks, so track index 0 falls through to the
|
||||
// downloaded-subtitle branch that queries the repository.
|
||||
file := &models.MediaFile{
|
||||
ID: 42,
|
||||
ContentID: "movie-1",
|
||||
FilePath: "/tmp/movie.mkv",
|
||||
Duration: 3600,
|
||||
}
|
||||
baseMgr := playback.NewSessionManager(0, 0)
|
||||
session, err := baseMgr.StartSession(1, "profile-1", 42, playback.PlayDirect, false)
|
||||
if err != nil {
|
||||
t.Fatalf("StartSession: %v", err)
|
||||
}
|
||||
|
||||
handler := NewStreamHandler(baseMgr, testPlaybackFileResolver{file: file})
|
||||
handler.SubtitleRepo = &handlerMockSubtitleRepo{listErr: errors.New("db unavailable")}
|
||||
handler.S3Client = newMockS3ClientForHandler()
|
||||
handler.S3Bucket = "test-bucket"
|
||||
|
||||
req := httptest.NewRequest(http.MethodGet, "/api/v1/stream/"+session.ID+"/subtitles/0.vtt", nil)
|
||||
req = req.WithContext(newAuthorizedPlaybackContext())
|
||||
routeCtx := chi.NewRouteContext()
|
||||
routeCtx.URLParams.Add("session_id", session.ID)
|
||||
routeCtx.URLParams.Add("track", "0.vtt")
|
||||
req = req.WithContext(context.WithValue(req.Context(), chi.RouteCtxKey, routeCtx))
|
||||
|
||||
rr := httptest.NewRecorder()
|
||||
handler.HandleSubtitle(rr, req)
|
||||
|
||||
if rr.Code != http.StatusInternalServerError {
|
||||
t.Fatalf("status = %d, body = %s", rr.Code, rr.Body.String())
|
||||
}
|
||||
var body struct {
|
||||
Error string `json:"error"`
|
||||
}
|
||||
if err := json.Unmarshal(rr.Body.Bytes(), &body); err != nil {
|
||||
t.Fatalf("decode error body: %v (body = %s)", err, rr.Body.String())
|
||||
}
|
||||
if body.Error != "internal_error" {
|
||||
t.Fatalf("error code = %q, want %q (body = %s)", body.Error, "internal_error", rr.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleTransportStartFailure_KeepsSessionForNonMissingError(t *testing.T) {
|
||||
filePath := writePlaybackTestMediaFile(t, "movie.mkv")
|
||||
file := &models.MediaFile{
|
||||
|
||||
@@ -239,6 +239,9 @@ type handlerMockSubtitleRepo struct {
|
||||
subtitles map[int]*subtitles.DownloadedSubtitle
|
||||
nextID int
|
||||
byKey map[string]*subtitles.DownloadedSubtitle
|
||||
// listErr, when set, is returned by ListDownloadedSubtitles to simulate a
|
||||
// backing-store failure.
|
||||
listErr error
|
||||
}
|
||||
|
||||
func newMockSubtitleRepoForHandler() *handlerMockSubtitleRepo {
|
||||
@@ -265,6 +268,9 @@ func (m *handlerMockSubtitleRepo) GetDownloadedSubtitle(_ context.Context, id in
|
||||
}
|
||||
|
||||
func (m *handlerMockSubtitleRepo) ListDownloadedSubtitles(context.Context, int) ([]subtitles.DownloadedSubtitle, error) {
|
||||
if m.listErr != nil {
|
||||
return nil, m.listErr
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"github.com/go-chi/chi/v5"
|
||||
|
||||
apimw "github.com/Silo-Server/silo-server/internal/api/middleware"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/collections/templates"
|
||||
"github.com/Silo-Server/silo-server/internal/mdblist"
|
||||
"github.com/Silo-Server/silo-server/internal/s3client"
|
||||
@@ -65,13 +66,14 @@ func (h *UserCollectionImportHandler) HandleListTemplates(w http.ResponseWriter,
|
||||
}
|
||||
|
||||
type userImportSharedFields struct {
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
Limit *int `json:"limit,omitempty"`
|
||||
SyncSchedule string `json:"sync_schedule"`
|
||||
IsShared bool `json:"is_shared"`
|
||||
PosterURL string `json:"poster_url"`
|
||||
LibraryIDs []int `json:"library_ids,omitempty"`
|
||||
Title string `json:"title"`
|
||||
Description string `json:"description"`
|
||||
Limit *int `json:"limit,omitempty"`
|
||||
SyncSchedule string `json:"sync_schedule"`
|
||||
IsShared bool `json:"is_shared"`
|
||||
PosterURL string `json:"poster_url"`
|
||||
LibraryIDs []int `json:"library_ids,omitempty"`
|
||||
DisplayQueryDefinition json.RawMessage `json:"display_query_definition,omitempty"`
|
||||
}
|
||||
|
||||
type userImportMDBListRequest struct {
|
||||
@@ -200,26 +202,35 @@ func (h *UserCollectionImportHandler) createImportedCollection(
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
if !validateOptionalLibraryIDs(cfg.LibraryIDs, w) {
|
||||
return
|
||||
}
|
||||
|
||||
sourceConfigJSON, err := usercollections.MarshalSourceConfig(cfg)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to encode source config")
|
||||
return
|
||||
}
|
||||
displayQueryDefinition, err := catalog.NormalizeDisplayQueryFragment(shared.DisplayQueryDefinition)
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
||||
return
|
||||
}
|
||||
|
||||
collection, err := store.CreateCollection(r.Context(), userstore.CreateCollectionInput{
|
||||
CreatorProfileID: profileID,
|
||||
Name: strings.TrimSpace(shared.Title),
|
||||
Description: strings.TrimSpace(shared.Description),
|
||||
CollectionType: collectionType,
|
||||
IsShared: shared.IsShared,
|
||||
QueryDefinition: "{}",
|
||||
SortConfig: "{}",
|
||||
SourceURL: cfg.DisplayURL(),
|
||||
SourceConfig: sourceConfigJSON,
|
||||
SyncSchedule: schedule,
|
||||
NextSyncAt: usercollections.InitialNextSyncAt(schedule),
|
||||
PosterURL: strings.TrimSpace(shared.PosterURL),
|
||||
CreatorProfileID: profileID,
|
||||
Name: strings.TrimSpace(shared.Title),
|
||||
Description: strings.TrimSpace(shared.Description),
|
||||
CollectionType: collectionType,
|
||||
IsShared: shared.IsShared,
|
||||
QueryDefinition: "{}",
|
||||
SortConfig: "{}",
|
||||
SourceURL: cfg.DisplayURL(),
|
||||
SourceConfig: sourceConfigJSON,
|
||||
SyncSchedule: schedule,
|
||||
NextSyncAt: usercollections.InitialNextSyncAt(schedule),
|
||||
DisplayQueryDefinition: displayQueryDefinition,
|
||||
PosterURL: strings.TrimSpace(shared.PosterURL),
|
||||
})
|
||||
if err != nil {
|
||||
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to create collection")
|
||||
@@ -425,3 +436,13 @@ func validateOptionalLimit(limit *int, w http.ResponseWriter) bool {
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func validateOptionalLibraryIDs(libraryIDs []int, w http.ResponseWriter) bool {
|
||||
for _, id := range libraryIDs {
|
||||
if id <= 0 {
|
||||
writeError(w, http.StatusBadRequest, "bad_request", "library_ids must contain positive IDs")
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -123,7 +123,7 @@ func resolveItemUserStatesWithOptions(
|
||||
}
|
||||
}
|
||||
|
||||
progressMap, err := store.ListProgressByMediaItems(ctx, profileID, progressIDs)
|
||||
progressMap, err := userstore.ListProgressWithCompletedHistory(ctx, store, profileID, progressIDs)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ import (
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
"github.com/Silo-Server/silo-server/internal/userstore"
|
||||
)
|
||||
|
||||
func TestResolveItemUserStatesIncludesCompletedEbookReaderProgress(t *testing.T) {
|
||||
@@ -94,3 +95,32 @@ func TestResolveItemUserStatesExcludesHiddenEbookReaderProgress(t *testing.T) {
|
||||
t.Fatalf("ebook updated after hidden_before = %+v, want played again", states["ebook-reread"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveItemUserStatesIncludesCompletedHistory(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
store := newProfileTestStore(t)
|
||||
addCompletedHistoryForUserDataTest(t, store, "movie-history-only")
|
||||
items := []*models.MediaItem{
|
||||
{ContentID: "movie-history-only", Type: "movie", Title: "Imported Movie"},
|
||||
}
|
||||
|
||||
states, err := resolveItemUserStates(ctx, store, "profile-1", nil, items)
|
||||
if err != nil {
|
||||
t.Fatalf("resolveItemUserStates: %v", err)
|
||||
}
|
||||
if states["movie-history-only"] == nil || !states["movie-history-only"].Played {
|
||||
t.Fatalf("history-only movie state = %+v, want played", states["movie-history-only"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAllEpisodesCompletedIncludesCompletedHistory(t *testing.T) {
|
||||
episodes := []*models.Episode{{ContentID: "episode-progress"}, {ContentID: "episode-history"}}
|
||||
progress := map[string]userstore.WatchProgress{
|
||||
"episode-progress": {MediaItemID: "episode-progress", Completed: true},
|
||||
"episode-history": {MediaItemID: "episode-history", Completed: true},
|
||||
}
|
||||
|
||||
if !allEpisodesCompleted(episodes, progress) {
|
||||
t.Fatal("allEpisodesCompleted = false, want completed from progress plus history")
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"io"
|
||||
"net/http"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/access"
|
||||
@@ -110,9 +111,16 @@ type watchTogetherPingMessage struct {
|
||||
ClientSentAt string `json:"client_sent_at"`
|
||||
}
|
||||
|
||||
// watchTogetherRoomConn serializes every write to the underlying gorilla
|
||||
// connection. gorilla/websocket does not support concurrent writers, and room
|
||||
// broadcasts arrive from other members' goroutines, so all writes — including
|
||||
// pong/error replies from the read loop — must go through this wrapper.
|
||||
type watchTogetherRoomConn struct {
|
||||
conn *websocket.Conn
|
||||
writeMu sync.Mutex
|
||||
// pingSentAtNano is the send time of the most recent protocol-level ping,
|
||||
// used to measure round-trip latency when the pong arrives.
|
||||
pingSentAtNano atomic.Int64
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) WriteJSON(v any) error {
|
||||
@@ -130,9 +138,35 @@ func (c *watchTogetherRoomConn) Close() error {
|
||||
func (c *watchTogetherRoomConn) WritePing() error {
|
||||
c.writeMu.Lock()
|
||||
defer c.writeMu.Unlock()
|
||||
c.pingSentAtNano.Store(time.Now().UnixNano())
|
||||
return writeWebSocketControl(c.conn, websocket.PingMessage, nil)
|
||||
}
|
||||
|
||||
// TakePingSentAt returns and clears the send time of the last unanswered
|
||||
// protocol-level ping.
|
||||
func (c *watchTogetherRoomConn) TakePingSentAt() time.Time {
|
||||
nano := c.pingSentAtNano.Swap(0)
|
||||
if nano == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
return time.Unix(0, nano)
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) WriteError(code, message string) {
|
||||
_ = c.WriteJSON(map[string]string{
|
||||
"type": "error",
|
||||
"code": code,
|
||||
"message": message,
|
||||
})
|
||||
}
|
||||
|
||||
func (c *watchTogetherRoomConn) writeRoomClosed(reason string) {
|
||||
_ = c.WriteJSON(map[string]string{
|
||||
"type": "room_closed",
|
||||
"reason": reason,
|
||||
})
|
||||
}
|
||||
|
||||
func NewWatchTogetherHandler(
|
||||
service *watchtogether.Service,
|
||||
scopeResolver WatchTogetherScopeResolver,
|
||||
@@ -653,19 +687,33 @@ func (h *WatchTogetherHandler) HandleRoomWebSocket(w http.ResponseWriter, r *htt
|
||||
|
||||
reg, snapshot, err := h.Service.Connect(ctx, roomID, claims.UserID, profileID, realtimeConn)
|
||||
if err != nil {
|
||||
// Terminal failures use room_closed so clients stop reconnecting
|
||||
// instead of retrying a room that will never come back.
|
||||
if errors.Is(err, watchtogether.ErrRoomNotFound) {
|
||||
writeWebSocketError(conn, "not_found", "Room not found")
|
||||
realtimeConn.writeRoomClosed("not_found")
|
||||
} else if errors.Is(err, watchtogether.ErrRoomClosed) {
|
||||
writeWebSocketError(conn, "gone", "Room is no longer active")
|
||||
realtimeConn.writeRoomClosed("ended")
|
||||
} else {
|
||||
writeWebSocketError(conn, "internal_error", "Failed to connect room socket")
|
||||
realtimeConn.WriteError("internal_error", "Failed to connect room socket")
|
||||
}
|
||||
return
|
||||
}
|
||||
defer h.Service.Disconnect(reg, false)
|
||||
|
||||
configureWebSocket(conn)
|
||||
// Measure round-trip latency from protocol-level ping/pong on the server
|
||||
// clock; client-reported timestamps are subject to clock skew and cannot
|
||||
// be trusted for command scheduling.
|
||||
conn.SetPongHandler(func(string) error {
|
||||
_ = conn.SetReadDeadline(time.Now().Add(wsPingInterval + wsPongTimeout))
|
||||
if sentAt := realtimeConn.TakePingSentAt(); !sentAt.IsZero() {
|
||||
_ = h.Service.HandlePingForConnection(ctx, reg, claims.UserID, profileID, time.Since(sentAt).Milliseconds())
|
||||
}
|
||||
return nil
|
||||
})
|
||||
startWebSocketPingLoop(ctx, realtimeConn.WritePing)
|
||||
// Prime an RTT sample right away instead of waiting for the first tick.
|
||||
_ = realtimeConn.WritePing()
|
||||
|
||||
if err := realtimeConn.WriteJSON(map[string]any{
|
||||
"type": "snapshot",
|
||||
@@ -680,15 +728,15 @@ func (h *WatchTogetherHandler) HandleRoomWebSocket(w http.ResponseWriter, r *htt
|
||||
return
|
||||
}
|
||||
|
||||
if err := h.handleRoomClientMessage(ctx, conn, reg, claims.UserID, profileID, data); err != nil {
|
||||
writeWebSocketError(conn, "bad_request", err.Error())
|
||||
if err := h.handleRoomClientMessage(ctx, realtimeConn, reg, claims.UserID, profileID, data); err != nil {
|
||||
realtimeConn.WriteError("bad_request", err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (h *WatchTogetherHandler) handleRoomClientMessage(
|
||||
ctx context.Context,
|
||||
conn *websocket.Conn,
|
||||
rc *watchTogetherRoomConn,
|
||||
reg *watchtogether.Registration,
|
||||
userID int,
|
||||
profileID string,
|
||||
@@ -768,16 +816,11 @@ func (h *WatchTogetherHandler) handleRoomClientMessage(
|
||||
if err := json.Unmarshal(data, &msg); err != nil {
|
||||
return err
|
||||
}
|
||||
clientSentAt, err := time.Parse(time.RFC3339Nano, msg.ClientSentAt)
|
||||
if err != nil {
|
||||
return errors.New("client_sent_at must be RFC3339Nano")
|
||||
}
|
||||
// The echoed timestamps only serve the client's clock-offset estimate;
|
||||
// latency for command scheduling is measured server-side from
|
||||
// protocol-level ping/pong.
|
||||
now := time.Now().UTC()
|
||||
pingMS := now.Sub(clientSentAt.UTC()).Milliseconds()
|
||||
if pingMS > 0 {
|
||||
_ = h.Service.HandlePingForConnection(ctx, reg, userID, profileID, pingMS)
|
||||
}
|
||||
return writeWebSocketJSON(conn, map[string]string{
|
||||
return rc.WriteJSON(map[string]string{
|
||||
"type": "pong",
|
||||
"client_sent_at": msg.ClientSentAt,
|
||||
"server_received_at": now.Format(time.RFC3339Nano),
|
||||
|
||||
+126
-11
@@ -32,11 +32,12 @@ import (
|
||||
"github.com/Silo-Server/silo-server/internal/catalogseed"
|
||||
"github.com/Silo-Server/silo-server/internal/clientip"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/download"
|
||||
"github.com/Silo-Server/silo-server/internal/downloads"
|
||||
evt "github.com/Silo-Server/silo-server/internal/events"
|
||||
"github.com/Silo-Server/silo-server/internal/historyimport"
|
||||
"github.com/Silo-Server/silo-server/internal/intromarkers"
|
||||
"github.com/Silo-Server/silo-server/internal/libraryingest"
|
||||
"github.com/Silo-Server/silo-server/internal/literaryworks"
|
||||
"github.com/Silo-Server/silo-server/internal/logstream"
|
||||
"github.com/Silo-Server/silo-server/internal/mail"
|
||||
"github.com/Silo-Server/silo-server/internal/markers"
|
||||
@@ -113,6 +114,7 @@ type Dependencies struct {
|
||||
AdminStatsProvider handlers.AdminStatsSource
|
||||
Recommender recommendations.Recommender // nil when disabled
|
||||
RecWorker *recommendations.Worker // nil when disabled
|
||||
CatalogSearchVectorizer catalog.CatalogSearchQueryVectorizer
|
||||
RatingsRepo *catalog.RatingsRepo
|
||||
PersonRepo *catalog.PersonRepository
|
||||
PersonRefreshQueue handlers.PersonRefreshQueue
|
||||
@@ -130,8 +132,9 @@ type Dependencies struct {
|
||||
ActivityLogRepo *activitylog.Repo
|
||||
OpsLogRepo *opslog.Repo
|
||||
FFmpegLogSink playback.FFmpegLogSink
|
||||
RedisClient *redis.Client // for session listing (may be nil)
|
||||
TaskManager *taskmanager.TaskManager // task manager (may be nil)
|
||||
RedisClient *redis.Client // for session listing (may be nil)
|
||||
TaskManager *taskmanager.TaskManager // task manager (may be nil)
|
||||
ArtifactManager *downloads.ArtifactManager // download prepare-to-file pipeline (may be nil)
|
||||
AdminJobCancelRegistry *adminjob.CancelRegistry
|
||||
IntroRepository *intromarkers.Repository
|
||||
IntroAnalyzer *intromarkers.Analyzer
|
||||
@@ -141,6 +144,7 @@ type Dependencies struct {
|
||||
MarkerContributionStore *markers.ContributionStore
|
||||
MarkerContributionService *markers.ContributionService
|
||||
WatchProviderService handlers.WatchProviderService
|
||||
WatchCompletionObserver watchstate.CompletionObserver
|
||||
PluginService *plugins.Service
|
||||
PluginHTTPProxy *plugins.HTTPProxy
|
||||
PluginUserConfig *plugins.UserConfigStore
|
||||
@@ -424,6 +428,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
var itemsHandler *handlers.ItemsHandler
|
||||
var catalogResourceHandler *handlers.CatalogResourceHandler
|
||||
var catalogHandler *handlers.CatalogHandler
|
||||
var literaryWorkHandler *handlers.LiteraryWorkHandler
|
||||
var peopleHandler *handlers.PeopleHandler
|
||||
var itemRepo *catalog.ItemRepository
|
||||
var episodeRepo *catalog.EpisodeRepository
|
||||
@@ -431,6 +436,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
var seasonRepo *catalog.SeasonRepository
|
||||
var detailSvc *catalog.DetailService
|
||||
var calendarRepo *catalog.CalendarRepository
|
||||
var catalogSearchService *catalog.CatalogSearchService
|
||||
var webhookSyncHandler *handlers.WebhookSyncHandler
|
||||
var requestHandler *handlers.RequestsHandler
|
||||
var autoscanHandler *handlers.AutoscanHandler
|
||||
@@ -444,6 +450,23 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
ebookAnnotationStore = handlers.NewPGEbookReaderAnnotationStore(deps.DB)
|
||||
browseRepo := catalog.NewBrowseRepository(deps.DB)
|
||||
itemRepo = catalog.NewItemRepository(deps.DB)
|
||||
searchIndexEvents := catalog.NewSearchIndexEventRepository(deps.DB)
|
||||
catalogSearchService = catalog.NewCatalogSearchService(
|
||||
context.Background(),
|
||||
settingsRepo,
|
||||
itemRepo,
|
||||
searchIndexEvents,
|
||||
deps.CatalogSearchVectorizer,
|
||||
)
|
||||
if catalogSearchService != nil {
|
||||
catalogSearchService.StartCoverageRefresh(deps.AppContext)
|
||||
}
|
||||
activeSearchProvider := catalog.SearchProviderPostgres
|
||||
if _, ok := catalogSearchService.Provider().(*catalog.MeilisearchSearchProvider); ok {
|
||||
activeSearchProvider = catalog.SearchProviderMeilisearch
|
||||
}
|
||||
searchIndexEvents.WithActiveProvider(activeSearchProvider)
|
||||
itemRepo.WithSearchIndexEvents(searchIndexEvents)
|
||||
episodeRepo = catalog.NewEpisodeRepository(deps.DB)
|
||||
providerIDRepo = catalog.NewProviderIDRepository(deps.DB)
|
||||
calendarRepo = catalog.NewCalendarRepository(deps.DB)
|
||||
@@ -463,10 +486,13 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
|
||||
rootClaimRepo := catalog.NewRootClaimRepository(deps.DB)
|
||||
groupClaimRepo := catalog.NewGroupClaimRepository(deps.DB)
|
||||
literaryRepo := literaryworks.NewRepository(deps.DB)
|
||||
literaryWorkHandler = &handlers.LiteraryWorkHandler{Service: literaryworks.NewService(literaryRepo)}
|
||||
detailSvc = catalog.NewDetailService(itemRepo, episodeRepo, seasonRepo, deps.PersonRepo, fileFetcher)
|
||||
detailSvc.SetFolderRepository(folderRepo)
|
||||
detailSvc.SetRootClaimRepository(rootClaimRepo)
|
||||
detailSvc.SetGroupClaimRepository(groupClaimRepo)
|
||||
detailSvc.SetWorkSummaryProvider(literaryRepo)
|
||||
detailSvc.SetProbeEnsurer(deps.ProbeEnsurer)
|
||||
detailSvc.SetChapterThumbnailQueuer(deps.ChapterThumbnailQueuer)
|
||||
if deps.ImageResolver != nil {
|
||||
@@ -484,6 +510,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
detailSvc,
|
||||
providerIDRepo,
|
||||
)
|
||||
if catalogSearchService != nil {
|
||||
itemsHandler.SetCatalogSearchProvider(catalogSearchService.Provider())
|
||||
}
|
||||
itemsHandler.EventsHub = deps.EventsHub
|
||||
itemsHandler.UserRepo = userRepo
|
||||
if requester, ok := deps.MetadataService.(handlers.MetadataRefreshRequester); ok {
|
||||
@@ -492,6 +521,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalWatchEventDispatcher); ok {
|
||||
itemsHandler.SetLocalWatchEventDispatcher(dispatcher)
|
||||
}
|
||||
if deps.WatchCompletionObserver != nil {
|
||||
itemsHandler.SetCompletionObserver(deps.WatchCompletionObserver)
|
||||
}
|
||||
if ebookProgressStore != nil {
|
||||
itemsHandler.SetEbookReaderProgressStore(ebookProgressStore)
|
||||
}
|
||||
@@ -510,14 +542,19 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if ebookAnnotationStore != nil {
|
||||
ebookReaderHandler.AnnotationStore = ebookAnnotationStore
|
||||
}
|
||||
if conv := buildEbookConversion(deps, settingsRepo); conv != nil {
|
||||
ebookReaderHandler.Conversion = conv
|
||||
}
|
||||
}
|
||||
catalogResourceHandler = handlers.NewCatalogResourceHandler(itemsHandler)
|
||||
catalogHandler = handlers.NewCatalogHandler(
|
||||
catalog.NewCatalogResolver(browseRepo, itemRepo).
|
||||
WithEpisodeRepository(episodeRepo).
|
||||
WithUserStoreProvider(deps.UserStoreProvider),
|
||||
WithUserStoreProvider(deps.UserStoreProvider).
|
||||
WithSearchProvider(catalogSearchService.Provider()),
|
||||
itemsHandler,
|
||||
)
|
||||
catalogHandler.SetWorkSummaryProvider(literaryRepo)
|
||||
|
||||
tmdbAPIKey := ""
|
||||
if deps.Config != nil {
|
||||
@@ -602,8 +639,8 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
personalDataHandler.SetEpisodeRepo(episodeRepo)
|
||||
personalDataHandler.SetSeasonRepo(seasonRepo)
|
||||
personalDataHandler.EventsHub = deps.EventsHub
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalFavoriteEventDispatcher); ok {
|
||||
personalDataHandler.SetLocalFavoriteEventDispatcher(dispatcher)
|
||||
if dispatcher, ok := deps.WatchProviderService.(handlers.LocalListEventDispatcher); ok {
|
||||
personalDataHandler.SetLocalListEventDispatcher(dispatcher)
|
||||
}
|
||||
progressHandler = handlers.NewProgressHandler(deps.UserStoreProvider)
|
||||
progressHandler.EventsHub = deps.EventsHub
|
||||
@@ -698,6 +735,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
playbackHandler.StoreProvider = deps.UserStoreProvider
|
||||
}
|
||||
playbackHandler.StableIdentityResolver = watchstate.NewStableIdentityResolver(itemRepo, episodeRepo, providerIDRepo)
|
||||
playbackHandler.CompletionObserver = deps.WatchCompletionObserver
|
||||
if scrobbler, ok := deps.WatchProviderService.(handlers.PlaybackWatchScrobbler); ok {
|
||||
playbackHandler.WatchScrobbler = scrobbler
|
||||
}
|
||||
@@ -710,6 +748,13 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
playbackHandler.SessionSyncer = deps.SessionSyncer
|
||||
}
|
||||
if streamHandler != nil {
|
||||
// Share the playback handler's transcode/reconstruct manager so a
|
||||
// direct/remux stream can rebuild its session from the token recipe
|
||||
// after a restart (same manager, same SessionManager).
|
||||
streamHandler.TM = playbackHandler.TranscodeManager()
|
||||
if deps.Config != nil {
|
||||
streamHandler.JWTSecret = deps.Config.Auth.JWTSecret
|
||||
}
|
||||
streamHandler.AdminStore = playbackAdminStore
|
||||
streamHandler.EventsHub = deps.EventsHub
|
||||
streamHandler.SessionSyncer = deps.SessionSyncer
|
||||
@@ -778,9 +823,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
watchtogether.NewRepository(deps.DB),
|
||||
deps.SessionMgr,
|
||||
deps.FileRepo,
|
||||
playbackHandler.CommandDispatcher,
|
||||
watchtogether.NewCatalogSelectionResolver(detailSvc),
|
||||
watchtogether.NewSuggestionRepository(deps.DB),
|
||||
watchtogether.NewProfileNameResolver(deps.UserStoreProvider),
|
||||
),
|
||||
viewerResolver,
|
||||
roomTokenService,
|
||||
@@ -822,6 +867,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
adminHandler.BootstrapSensitiveConfigured = deps.BootstrapSensitiveConfigured
|
||||
adminHandler.BootstrapSensitiveValues = deps.BootstrapSensitiveValues
|
||||
adminHandler.RestartStatus = restartStatus
|
||||
adminHandler.CatalogSearchStatus = catalogSearchService
|
||||
if settingsRepo != nil {
|
||||
adminHandler.SettingsRepo = settingsRepo
|
||||
}
|
||||
@@ -1311,18 +1357,18 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
// Build download handler.
|
||||
var downloadHandler *handlers.DownloadHandler
|
||||
if deps.DB != nil && deps.FileRepo != nil && deps.Config != nil {
|
||||
downloadRepo := download.NewRepository(deps.DB)
|
||||
downloadBandwidth := download.NewBandwidthManager(
|
||||
downloadRepo := downloads.NewRepository(deps.DB)
|
||||
downloadBandwidth := downloads.NewBandwidthManager(
|
||||
deps.Config.Download.ServerBandwidthBPS,
|
||||
deps.Config.Download.UserBandwidthBPS,
|
||||
)
|
||||
downloadLimiter := download.NewQuantityLimiter(
|
||||
downloadLimiter := downloads.NewQuantityLimiter(
|
||||
downloadRepo,
|
||||
deps.Config.Download.MaxConcurrentPerUser,
|
||||
deps.Config.Download.MaxPerPeriod,
|
||||
deps.Config.Download.PeriodDuration,
|
||||
)
|
||||
downloadSvc := download.NewService(
|
||||
downloadSvc := downloads.NewService(
|
||||
downloadRepo,
|
||||
downloadBandwidth,
|
||||
downloadLimiter,
|
||||
@@ -1334,7 +1380,31 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
settingsRepo,
|
||||
&deps.Config.Download,
|
||||
)
|
||||
if detailSvc != nil {
|
||||
// Offline manifest + artwork/subtitle proxies (Phase 2). subtitleManager
|
||||
// may be nil when subtitles are unconfigured; pass a nil interface so the
|
||||
// downloaded-subtitle path reports unavailable instead of panicking.
|
||||
var subtitleSource downloads.SubtitleSource
|
||||
if subtitleManager != nil {
|
||||
subtitleSource = subtitleManager
|
||||
}
|
||||
downloadSvc.SetOfflineDeps(detailSvc, subtitleSource, nil)
|
||||
}
|
||||
if deps.ArtifactManager != nil {
|
||||
// Prepare-to-file pipeline (Phase 3): remux/transcode-to-single-file.
|
||||
downloadSvc.SetArtifactManager(deps.ArtifactManager)
|
||||
}
|
||||
// Series monitoring (auto-download subscriptions). Client-pull only:
|
||||
// devices sync on app open / background refresh; there is no server
|
||||
// background worker.
|
||||
downloadSvc.SetSubscriptions(downloads.NewSubscriptionRepository(deps.DB))
|
||||
downloadHandler = handlers.NewDownloadHandler(downloadSvc)
|
||||
if profileHandler != nil {
|
||||
// Profiles may live outside Postgres (sqlite userdb backend), so
|
||||
// deleting one cannot FK-cascade the shared user_devices table;
|
||||
// purge the device library (and its downloads) in-app instead.
|
||||
profileHandler.DeviceLibraryPurger = downloadRepo
|
||||
}
|
||||
} else {
|
||||
downloadHandler = handlers.NewDownloadHandler(nil)
|
||||
}
|
||||
@@ -1588,6 +1658,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
}
|
||||
notificationsHandler := handlers.NewNotificationsHandler(deps.Notifications, deps.EventsHub)
|
||||
r.With(apimw.RequireProfile).Post("/events/ws-ticket", notificationsHandler.HandleMintWSTicket)
|
||||
r.With(apimw.RequireProfile).Post("/devices/push/apple", notificationsHandler.HandleRegisterApplePushDevice)
|
||||
// Discord DM channel: the linked identity and mode hang off
|
||||
// the login account, not a profile, so these stay outside
|
||||
// the RequireProfile subrouter below (static paths coexist
|
||||
@@ -1606,6 +1677,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/capability", notificationsHandler.HandleCapability)
|
||||
r.Get("/preferences", notificationsHandler.HandleGetPreferences)
|
||||
r.Put("/preferences", notificationsHandler.HandleUpdatePreferences)
|
||||
r.Get("/push/apple/display/{delivery_id}", notificationsHandler.HandleApplePushDisplay)
|
||||
r.Get("/email-preferences", notificationsHandler.HandleGetEmailPreferences)
|
||||
r.Put("/email-preferences", notificationsHandler.HandleUpdateEmailPreferences)
|
||||
r.Put("/email-preferences/address", notificationsHandler.HandleRequestEmailAddress)
|
||||
@@ -1688,10 +1760,14 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/catalog/filters/search", catalogHandler.HandleGetCatalogFacetSearch)
|
||||
r.Get("/catalog/audiobook-groups", catalogHandler.HandleGetAudiobookGroups)
|
||||
r.Post("/catalog/query", catalogHandler.HandlePostCatalogQuery)
|
||||
if literaryWorkHandler != nil {
|
||||
r.Get("/works/{work_id}", literaryWorkHandler.HandleGetWork)
|
||||
}
|
||||
if catalogResourceHandler != nil {
|
||||
r.Get("/catalog/items/{id}", catalogResourceHandler.HandleGetItemDetail)
|
||||
r.Get("/catalog/items/{id}/episodes", catalogResourceHandler.HandleGetItemEpisodes)
|
||||
r.Get("/catalog/items/{id}/versions", catalogResourceHandler.HandleGetItemVersions)
|
||||
r.Get("/catalog/items/{id}/manga-files", catalogResourceHandler.HandleGetMangaFiles)
|
||||
r.Get("/catalog/series/{id}/seasons", catalogResourceHandler.HandleGetSeasons)
|
||||
r.Get("/catalog/series/{id}/seasons/{num}", catalogResourceHandler.HandleGetSeason)
|
||||
r.Get("/catalog/series/{id}/seasons/{num}/episodes", catalogResourceHandler.HandleGetEpisodes)
|
||||
@@ -1805,6 +1881,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Route("/collections", func(r chi.Router) {
|
||||
r.Use(apimw.RequireProfile)
|
||||
r.Get("/", collectionHandler.HandleListCollections)
|
||||
r.Get("/capabilities", collectionHandler.HandleCapabilities)
|
||||
if libraryCollectionHandler != nil {
|
||||
// Aggregated server (admin-curated) collections across
|
||||
// every accessible library. Separate from "/" (personal,
|
||||
@@ -1983,6 +2060,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if ebookReaderHandler != nil {
|
||||
r.Route("/ebooks", func(r chi.Router) {
|
||||
r.Use(apimw.RequireProfile)
|
||||
r.Get("/capability", ebookReaderHandler.HandleConversionCapability)
|
||||
r.Get("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
|
||||
r.Head("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
|
||||
r.Get("/{content_id}/progress", ebookReaderHandler.HandleGetProgress)
|
||||
@@ -2003,6 +2081,8 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/metadata/ai/status", metadataAIHandler.HandleStatus)
|
||||
if itemRepo != nil {
|
||||
metadataAIHandler.ItemAccess = itemRepo
|
||||
metadataAIHandler.SeasonLookup = seasonRepo
|
||||
metadataAIHandler.EpisodeLookup = episodeRepo
|
||||
r.Post("/items/{id}/translate-description", metadataAIHandler.HandleTranslateOnView)
|
||||
}
|
||||
} else {
|
||||
@@ -2105,10 +2185,27 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
|
||||
// Download routes.
|
||||
r.Route("/downloads", func(r chi.Router) {
|
||||
r.Get("/capability", downloadHandler.HandleCapability)
|
||||
r.Post("/", downloadHandler.HandleCreateDownload)
|
||||
r.Get("/", downloadHandler.HandleListDownloads)
|
||||
// Series monitoring (auto-download) subscriptions.
|
||||
r.Post("/subscriptions", downloadHandler.HandleCreateSubscription)
|
||||
r.Post("/subscriptions/sync", downloadHandler.HandleSyncSubscriptions)
|
||||
r.Get("/subscriptions", downloadHandler.HandleListSubscriptions)
|
||||
r.Get("/subscriptions/{id}", downloadHandler.HandleGetSubscription)
|
||||
r.Patch("/subscriptions/{id}", downloadHandler.HandlePatchSubscription)
|
||||
r.Delete("/subscriptions/{id}", downloadHandler.HandleDeleteSubscription)
|
||||
r.Get("/batches/{batch_id}/manifests", downloadHandler.HandleBatchManifests)
|
||||
r.Patch("/{id}", downloadHandler.HandlePatchDownload)
|
||||
r.Delete("/{id}", downloadHandler.HandleDeleteDownload)
|
||||
// GET+HEAD: background download stacks probe with HEAD
|
||||
// before issuing ranged GETs; http.ServeContent handles
|
||||
// HEAD natively.
|
||||
r.Get("/{id}/file", downloadHandler.HandleDownloadFile)
|
||||
r.Head("/{id}/file", downloadHandler.HandleDownloadFile)
|
||||
r.Get("/{id}/manifest", downloadHandler.HandleManifest)
|
||||
r.Get("/{id}/artwork/{kind}", downloadHandler.HandleArtwork)
|
||||
r.Get("/{id}/subtitles/{ref}", downloadHandler.HandleSubtitle)
|
||||
})
|
||||
r.Get("/direct-download", downloadHandler.HandleDirectDownload)
|
||||
r.Head("/direct-download", downloadHandler.HandleDirectDownload)
|
||||
@@ -2221,6 +2318,14 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
r.Get("/unmatched", adminHandler.HandleListUnmatched)
|
||||
r.Get("/stats", adminHandler.HandleGetStats)
|
||||
r.Get("/server/status", adminHandler.HandleGetServerStatus)
|
||||
r.Get("/catalog/search/status", adminHandler.HandleGetCatalogSearchStatus)
|
||||
if literaryWorkHandler != nil {
|
||||
r.Get("/literary-works/items/{content_id}/candidates", literaryWorkHandler.HandleListCandidates)
|
||||
r.Post("/literary-works/link", literaryWorkHandler.HandleLinkItems)
|
||||
r.Delete("/literary-works/{work_id}/items/{content_id}", literaryWorkHandler.HandleUnlinkItem)
|
||||
r.Post("/literary-works/matches/confirm", literaryWorkHandler.HandleConfirmMatch)
|
||||
r.Post("/literary-works/matches/ignore", literaryWorkHandler.HandleIgnoreMatch)
|
||||
}
|
||||
r.Post("/server/restart", serverControlHandler.HandleRestart)
|
||||
r.Get("/jellyfin-compat/status", adminHandler.HandleGetJellyfinCompatStatus)
|
||||
r.Patch("/jellyfin-compat/settings", adminHandler.HandleUpdateJellyfinCompatSettings)
|
||||
@@ -2249,6 +2354,15 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if discordNotificationsHandler != nil {
|
||||
r.Post("/notifications/discord/test", discordNotificationsHandler.HandleAdminTest)
|
||||
}
|
||||
if deps.Notifications != nil || settingsRepo != nil {
|
||||
applePushHandler := handlers.NewAdminApplePushHandler(deps.Notifications, settingsRepo)
|
||||
if deps.Notifications != nil {
|
||||
r.Post("/notifications/push/apple/test", applePushHandler.HandleTest)
|
||||
}
|
||||
if settingsRepo != nil {
|
||||
r.Post("/notifications/push/relay/register", applePushHandler.HandleRegisterRelay)
|
||||
}
|
||||
}
|
||||
if deps.Notifications != nil && deps.Notifications.ServerChannels != nil {
|
||||
serverChannelsHandler := handlers.NewAdminServerChannelsHandler(deps.Notifications)
|
||||
r.Route("/notifications/server-channels", func(r chi.Router) {
|
||||
@@ -2894,6 +3008,7 @@ func effectiveSubtitleAIConfig(cfg *config.Config) (subtitleai.Config, string) {
|
||||
ASRModel: cfg.AI.ASRModel,
|
||||
BatchSize: cfg.SubtitleAI.BatchSize,
|
||||
ContextNeighbors: cfg.SubtitleAI.ContextNeighbors,
|
||||
LiveASRChunkSeconds: cfg.SubtitleAI.LiveASRChunkSeconds,
|
||||
TranscribeQuotaJobs: cfg.SubtitleAI.TranscribeQuotaJobs,
|
||||
TranscribeQuotaPeriod: cfg.SubtitleAI.TranscribeQuotaPeriod,
|
||||
}, disabledGateway
|
||||
|
||||
@@ -174,6 +174,22 @@ func (s *stubMediaStore) GetAudiobookByID(_ context.Context, id string, _ catalo
|
||||
}
|
||||
return nil, nil
|
||||
}
|
||||
func (s *stubMediaStore) GetAudiobooksByIDs(_ context.Context, ids []string, _ catalog.AccessFilter) (map[string]*models.MediaItem, error) {
|
||||
if s.lookupErr != nil {
|
||||
return nil, s.lookupErr
|
||||
}
|
||||
out := make(map[string]*models.MediaItem, len(ids))
|
||||
for _, id := range ids {
|
||||
if it, ok := s.known[id]; ok {
|
||||
if it == nil {
|
||||
out[id] = &models.MediaItem{ContentID: id}
|
||||
} else {
|
||||
out[id] = it
|
||||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
func (s *stubMediaStore) GetAuthorByID(_ context.Context, id string, _ catalog.AccessFilter) (Author, error) {
|
||||
return Author{}, ErrNotFound
|
||||
}
|
||||
|
||||
@@ -172,6 +172,9 @@ func (h *Handler) handlePublicTrack(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "session expired", http.StatusGone)
|
||||
return
|
||||
}
|
||||
if h.beginNativePlaybackTransport(sid) {
|
||||
defer h.endNativePlaybackTransport(sid)
|
||||
}
|
||||
|
||||
access, err := h.accessFilterForAuth(r.Context(), ctxAuth{UserID: sess.UserID, ProfileID: sess.ProfileID})
|
||||
if err != nil {
|
||||
|
||||
@@ -41,6 +41,9 @@ type AudiobookLibrary struct {
|
||||
// a small adapter struct added in a later stage.
|
||||
type MediaStore interface {
|
||||
GetAudiobookByID(ctx context.Context, contentID string, access catalog.AccessFilter) (*models.MediaItem, error)
|
||||
// GetAudiobooksByIDs batch-fetches audiobooks by content_id (people + series
|
||||
// hydrated once), keyed by content_id, for list/shelf handlers.
|
||||
GetAudiobooksByIDs(ctx context.Context, contentIDs []string, access catalog.AccessFilter) (map[string]*models.MediaItem, error)
|
||||
// ListAudiobooks returns a page of audiobooks. When libraryID is non-zero
|
||||
// it filters to items in that media_folder; 0 means all audiobook items.
|
||||
ListAudiobooks(ctx context.Context, libraryID int64, limit, offset int, access catalog.AccessFilter) ([]*models.MediaItem, int, error)
|
||||
@@ -255,6 +258,14 @@ type Dependencies struct {
|
||||
// SocketIO is the Socket.io server mounted at /abs/socket.io/. May be nil;
|
||||
// the route is only registered when a non-nil value is supplied.
|
||||
SocketIO SocketIOServer
|
||||
// NativeSessions mirrors ABS playback into Silo's native playback session
|
||||
// manager so shared live-session views, limits, and stale-session cleanup
|
||||
// see Audiobookshelf-compatible clients. May be nil; ABS playback still
|
||||
// functions, but admin live-session visibility is unavailable.
|
||||
NativeSessions PlaybackSessionManager
|
||||
// NativeSessionSyncer flushes native session-manager state into the shared
|
||||
// admin live-session table after ABS play/sync/close events.
|
||||
NativeSessionSyncer PlaybackSessionSyncer
|
||||
// CoverResolver translates a raw silo poster path (e.g.
|
||||
// "local/audiobooks/.../original.webp") into a fully-qualified URL
|
||||
// the ABS client can fetch. Optional; when nil, /api/items/{id}/cover
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user