Files
silo-server/internal/ebookconvert/converter_test.go
e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171)
* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 13:09:55 -04:00

248 lines
8.0 KiB
Go

package ebookconvert
import (
"archive/zip"
"context"
"errors"
"io"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func newTestConverter(t *testing.T) *Converter {
t.Helper()
c, err := NewConverter(context.Background(), Options{})
if err != nil {
t.Fatalf("NewConverter: %v", err)
}
t.Cleanup(func() { _ = c.Close(context.Background()) })
return c
}
func TestConvert_DRMFreeProducesValidEpub(t *testing.T) {
for _, fixture := range []string{"sample-ncx.mobi", "sample-cp1252.mobi"} {
t.Run(fixture, func(t *testing.T) {
c := newTestConverter(t)
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(context.Background(), filepath.Join("testdata", fixture), dst); err != nil {
t.Fatalf("Convert: %v", err)
}
// Independently re-validate the delivered file is a real EPUB.
assertValidEpub(t, dst)
})
}
}
func TestConvert_DRMProtectedRejected(t *testing.T) {
c := newTestConverter(t)
dst := filepath.Join(t.TempDir(), "out.epub")
err := c.Convert(context.Background(), filepath.Join("testdata", "sample-drm-v1.mobi"), dst)
if !errors.Is(err, ErrDRMProtected) {
t.Fatalf("got %v, want ErrDRMProtected", err)
}
if _, statErr := os.Stat(dst); !os.IsNotExist(statErr) {
t.Fatalf("DRM source must not produce an output file, but %s exists", dst)
}
}
func TestConvert_OversizeRejected(t *testing.T) {
c, err := NewConverter(context.Background(), Options{MaxSourceBytes: 1024})
if err != nil {
t.Fatalf("NewConverter: %v", err)
}
t.Cleanup(func() { _ = c.Close(context.Background()) })
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrSourceTooLarge) {
t.Fatalf("got %v, want ErrSourceTooLarge", err)
}
}
func TestConvert_CorruptSourceFails(t *testing.T) {
c := newTestConverter(t)
bad := filepath.Join(t.TempDir(), "bad.mobi")
if err := os.WriteFile(bad, []byte("not a mobi file at all"), 0o644); err != nil {
t.Fatal(err)
}
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(context.Background(), bad, dst); !errors.Is(err, ErrConversionFailed) {
t.Fatalf("got %v, want ErrConversionFailed", err)
}
}
func TestConvert_MissingSourceFails(t *testing.T) {
c := newTestConverter(t)
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(context.Background(), "testdata/does-not-exist.mobi", dst); !errors.Is(err, ErrConversionFailed) {
t.Fatalf("got %v, want ErrConversionFailed", err)
}
}
func TestConvert_Concurrent(t *testing.T) {
c := newTestConverter(t)
const n = 6
errs := make(chan error, n)
for i := 0; i < n; i++ {
go func(i int) {
dst := filepath.Join(t.TempDir(), "c.epub")
errs <- c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
}(i)
}
for i := 0; i < n; i++ {
if err := <-errs; err != nil {
t.Fatalf("concurrent Convert: %v", err)
}
}
}
func TestConvert_ContextCancelled(t *testing.T) {
c := newTestConverter(t)
ctx, cancel := context.WithCancel(context.Background())
cancel() // already canceled before the conversion starts
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(ctx, filepath.Join("testdata", "sample-ncx.mobi"), dst); err == nil {
t.Fatal("expected error on canceled context")
}
}
func TestValidateEpub_RejectsNonEpubZip(t *testing.T) {
// A zip that is not an EPUB (no mimetype-first) must be rejected.
p := filepath.Join(t.TempDir(), "plain.zip")
f, err := os.Create(p)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
w, _ := zw.Create("hello.txt")
_, _ = w.Write([]byte("hi"))
_ = zw.Close()
_ = f.Close()
if err := validateEpub(p); err == nil {
t.Fatal("expected validateEpub to reject a non-EPUB zip")
}
}
func TestValidateEpub_RejectsDeflatedMimetype(t *testing.T) {
// mimetype present + correct content but DEFLATED (not stored) -> reject.
p := filepath.Join(t.TempDir(), "bad.epub")
f, err := os.Create(p)
if err != nil {
t.Fatal(err)
}
zw := zip.NewWriter(f)
w, _ := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Deflate})
_, _ = w.Write([]byte("application/epub+zip"))
_ = zw.Close()
_ = f.Close()
if err := validateEpub(p); err == nil {
t.Fatal("expected rejection of deflated mimetype")
}
}
func TestConvert_AfterCloseReturnsUnavailable(t *testing.T) {
c, err := NewConverter(context.Background(), Options{})
if err != nil {
t.Fatalf("NewConverter: %v", err)
}
_ = c.Close(context.Background())
dst := filepath.Join(t.TempDir(), "out.epub")
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrUnavailable) {
t.Fatalf("got %v, want ErrUnavailable", err)
}
}
func TestConvert_TimeoutClassifiedNotRawExit(t *testing.T) {
// A 1ns timeout must surface as ErrConversionTimedOut (a transient verdict,
// distinct from a deterministic failure), never as a bogus "mobitool exit
// <huge>" code.
c, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond})
if err != nil {
t.Fatalf("NewConverter: %v", err)
}
t.Cleanup(func() { _ = c.Close(context.Background()) })
dst := filepath.Join(t.TempDir(), "out.epub")
err = c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
if !errors.Is(err, ErrConversionTimedOut) {
t.Fatalf("got %v, want ErrConversionTimedOut", err)
}
if errors.Is(err, ErrConversionFailed) {
t.Fatalf("a timeout must not also satisfy ErrConversionFailed (it would be negatively cached): %v", err)
}
if strings.Contains(err.Error(), "exit ") {
t.Fatalf("timeout leaked a raw exit code: %v", err)
}
}
// classifyRunError must propagate a caller's deadline as context.DeadlineExceeded
// (CR review): reclassifying it as a conversion failure breaks upstream
// cancellation handling and would poison the negative cache.
func TestClassifyRunError_ParentDeadlinePropagates(t *testing.T) {
parent, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
defer cancel()
run := parent // run derives from parent, so it is also past-deadline
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
if !errors.Is(err, context.DeadlineExceeded) {
t.Fatalf("got %v, want context.DeadlineExceeded propagated", err)
}
if errors.Is(err, ErrConversionFailed) || errors.Is(err, ErrConversionTimedOut) {
t.Fatalf("a caller deadline must not be reclassified as a conversion verdict: %v", err)
}
}
// When the caller's context is healthy but our own per-call timeout fired, the
// result is the transient ErrConversionTimedOut sentinel (not ErrConversionFailed).
func TestClassifyRunError_OwnTimeoutIsTimedOutSentinel(t *testing.T) {
parent := context.Background() // caller healthy
run, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
defer cancel()
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
if !errors.Is(err, ErrConversionTimedOut) {
t.Fatalf("got %v, want ErrConversionTimedOut", err)
}
if errors.Is(err, ErrConversionFailed) {
t.Fatalf("own timeout must not satisfy ErrConversionFailed: %v", err)
}
}
func assertValidEpub(t *testing.T, path string) {
t.Helper()
zr, err := zip.OpenReader(path)
if err != nil {
t.Fatalf("open epub: %v", err)
}
defer zr.Close()
if len(zr.File) == 0 || zr.File[0].Name != "mimetype" {
t.Fatalf("first entry not mimetype")
}
rc, _ := zr.File[0].Open()
mt, _ := io.ReadAll(rc)
rc.Close()
if strings.TrimSpace(string(mt)) != "application/epub+zip" {
t.Fatalf("bad mimetype %q", mt)
}
var hasContainer, hasContent bool
for _, fl := range zr.File {
if fl.Name == "META-INF/container.xml" {
hasContainer = true
}
if strings.HasPrefix(fl.Name, "OEBPS/") {
hasContent = true
}
}
if !hasContainer {
t.Fatal("missing META-INF/container.xml")
}
if !hasContent {
t.Fatal("no OEBPS content")
}
}
// Guard so the default timeout constant stays sane if edited.
func TestDefaults(t *testing.T) {
if DefaultTimeout < time.Second {
t.Fatal("DefaultTimeout too small")
}
}