Files
silo-server/internal/api/handlers/ebook_convert_serve_test.go
e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171)
* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 13:09:55 -04:00

272 lines
9.3 KiB
Go

package handlers
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/Silo-Server/silo-server/internal/ebookconvert"
"github.com/Silo-Server/silo-server/internal/models"
)
type fakeConverter struct {
epubPath string
err error
calls int
lookupPath string
lookupErr error
lookupOK bool
}
func (f *fakeConverter) GetOrConvert(_ context.Context, _ string, _ ebookconvert.SourceKey) (string, error) {
f.calls++
return f.epubPath, f.err
}
func (f *fakeConverter) Lookup(_ ebookconvert.SourceKey) (string, error, bool) {
return f.lookupPath, f.lookupErr, f.lookupOK
}
func writeTemp(t *testing.T, name, content string) string {
t.Helper()
p := filepath.Join(t.TempDir(), name)
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
return p
}
func serveAndRecord(t *testing.T, h *EbookReaderHandler, file *models.MediaFile) *httptest.ResponseRecorder {
t.Helper()
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook: %v", err)
}
return rec
}
func enabledConversion(conv EbookConverter, on bool) *EbookConversion {
return &EbookConversion{Converter: conv, Enabled: func(context.Context) bool { return on }}
}
func TestServeEbook_ConvertsKindleWhenEnabled(t *testing.T) {
epub := writeTemp(t, "out.epub", "PK\x03\x04 fake-epub-bytes")
src := writeTemp(t, "book.azw3", "raw-azw3-source")
conv := &fakeConverter{epubPath: epub}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3", FileHash: "abc123"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
}
if rec.Header().Get("ETag") == "" {
t.Fatal("expected an ETag on converted response")
}
if rec.Body.String() != "PK\x03\x04 fake-epub-bytes" {
t.Fatalf("body = %q, want the epub bytes", rec.Body.String())
}
if conv.calls != 1 {
t.Fatalf("converter called %d times, want 1", conv.calls)
}
}
func TestServeEbook_ConversionFailureFallsBackToRawWithHeader(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
conv := &fakeConverter{err: ebookconvert.ErrDRMProtected}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 2, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get(ConversionHeader); got != "failed" {
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
}
if rec.Body.String() != "raw-mobi-bytes" {
t.Fatalf("body = %q, want raw mobi bytes", rec.Body.String())
}
}
func TestServeEbook_DisabledServesRaw(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
conv := &fakeConverter{epubPath: "should-not-be-used"}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, false)}
file := &models.MediaFile{ID: 3, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if got := rec.Header().Get(ConversionHeader); got != "" {
t.Fatalf("%s = %q, want empty (no conversion attempted)", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
}
if conv.calls != 0 {
t.Fatalf("converter called %d times while disabled, want 0", conv.calls)
}
}
func TestServeEbook_NonKindleUntouched(t *testing.T) {
raw := writeTemp(t, "book.epub", "epub-bytes")
conv := &fakeConverter{epubPath: "should-not-be-used"}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 4, FilePath: raw, Container: "epub"}
rec := serveAndRecord(t, h, file)
if rec.Header().Get(ConversionHeader) != "" {
t.Fatal("non-kindle file must not trigger conversion")
}
if conv.calls != 0 {
t.Fatalf("converter called %d times for epub, want 0", conv.calls)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want epub", got)
}
}
func TestServeEbook_NilConversionServesRaw(t *testing.T) {
raw := writeTemp(t, "book.mobi", "raw")
h := &EbookReaderHandler{} // Conversion nil
file := &models.MediaFile{ID: 5, FilePath: raw, Container: "mobi"}
rec := serveAndRecord(t, h, file)
if rec.Header().Get(ConversionHeader) != "" {
t.Fatal("nil Conversion must not set the conversion header")
}
}
func TestHandleConversionCapability(t *testing.T) {
for _, tc := range []struct {
name string
conv *EbookConversion
want bool
}{
{"enabled", enabledConversion(&fakeConverter{}, true), true},
{"flag-off", enabledConversion(&fakeConverter{}, false), false},
{"not-wired", nil, false},
} {
t.Run(tc.name, func(t *testing.T) {
h := &EbookReaderHandler{Conversion: tc.conv}
rec := httptest.NewRecorder()
h.HandleConversionCapability(rec, httptest.NewRequest(http.MethodGet, "/ebooks/capability", nil))
if rec.Code != http.StatusOK {
t.Fatalf("status = %d", rec.Code)
}
body := rec.Body.String()
wantField := `"enabled":false`
if tc.want {
wantField = `"enabled":true`
}
if !strings.Contains(body, wantField) {
t.Fatalf("body %q missing %s", body, wantField)
}
if !strings.Contains(body, `"served_format":"epub"`) || !strings.Contains(body, "mobi") {
t.Fatalf("capability body incomplete: %s", body)
}
})
}
}
// A HEAD on a cache miss must NOT trigger a (possibly minute-long) conversion;
// it advertises the converted representation cheaply and lets the GET do the work.
func TestServeEbook_HeadDoesNotConvertOnMiss(t *testing.T) {
src := writeTemp(t, "book.azw3", "raw-azw3-source")
conv := &fakeConverter{lookupOK: false} // cache miss
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a cache miss triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted (advertised on HEAD miss)", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
if rec.Header().Get("ETag") == "" {
t.Fatal("HEAD miss should still carry the conversion ETag")
}
}
// A HEAD that hits the cache serves the converted headers from the cached file,
// without converting.
func TestServeEbook_HeadServesCachedConverted(t *testing.T) {
epub := writeTemp(t, "out.epub", "PK\x03\x04 cached-epub")
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{lookupPath: epub, lookupOK: true}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 2, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a cache hit triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "converted" {
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
}
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
}
}
// A HEAD on a negatively-cached (DRM/failed) source reports the failed contract
// without converting.
func TestServeEbook_HeadDRMReturnsFailed(t *testing.T) {
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{lookupErr: ebookconvert.ErrDRMProtected}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 3, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodHead, "/read", nil)
if err := h.serveEbook(rec, req, file); err != nil {
t.Fatalf("serveEbook HEAD: %v", err)
}
if conv.calls != 0 {
t.Fatalf("HEAD on a negatively-cached source triggered %d conversions, want 0", conv.calls)
}
if got := rec.Header().Get(ConversionHeader); got != "failed" {
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
}
}
func TestServeEbook_ContextCancelPropagates(t *testing.T) {
src := writeTemp(t, "book.mobi", "raw-mobi-source")
conv := &fakeConverter{err: context.Canceled}
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
file := &models.MediaFile{ID: 6, FilePath: src, Container: "mobi"}
rec := httptest.NewRecorder()
req := httptest.NewRequest(http.MethodGet, "/read", nil)
err := h.serveEbook(rec, req, file)
if err == nil {
t.Fatal("expected context cancellation to propagate, not fall back")
}
if rec.Header().Get(ConversionHeader) == "failed" {
t.Fatal("cancellation must not be reported as a conversion failure")
}
}