Files
silo-server/internal/api/ebook_conversion.go
e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171)
* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 13:09:55 -04:00

104 lines
3.8 KiB
Go
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
package api
import (
"context"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"time"
"github.com/Silo-Server/silo-server/internal/api/handlers"
"github.com/Silo-Server/silo-server/internal/catalog"
"github.com/Silo-Server/silo-server/internal/config"
"github.com/Silo-Server/silo-server/internal/ebookconvert"
)
// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB
// conversion. Off by default; toggled via PUT /admin/settings/{key}.
const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled"
// buildEbookConversion wires the in-process Kindle->EPUB converter for the read
// handler. It compiles the embedded WASM module once at startup; if that fails
// the feature stays off (returns nil) and the raw original is served as before.
// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be
// toggled without a restart yet does not hit the DB on every read.
//
// The Converter (a wazero runtime, ~no external resources — per-conversion
// scratch dirs are cleaned as they go) is intentionally owned for the process
// lifetime: it is not Close()d on shutdown because process exit reclaims it and
// there is nothing to flush. Memory is bounded per conversion by
// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's
// resource notes); tune via ebookconvert.Options if the deployment is tight.
func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion {
if settings == nil {
return nil
}
converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{})
if err != nil {
slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err)
return nil
}
cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())})
if err != nil {
slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err)
_ = converter.Close(context.Background())
return nil
}
slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey)
return &handlers.EbookConversion{
Converter: cache,
Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey),
}
}
// ebookFlagPredicate returns a cached boolean reader for a server setting. The
// read path (every ebook read + the capability endpoint, which clients may
// poll) would otherwise issue a DB query per request; a short TTL collapses
// bursts to one query while keeping admin toggles effectively instant. On a read
// error it keeps the last known value rather than flapping the feature off.
func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool {
const ttl = 3 * time.Second
var (
mu sync.Mutex
value bool
fetched time.Time
hasVal bool
)
return func(ctx context.Context) bool {
mu.Lock()
defer mu.Unlock()
if hasVal && time.Since(fetched) < ttl {
return value
}
v, err := settings.Get(ctx, key)
if err != nil {
return value // keep last known (false until the first successful read)
}
value = isTruthySetting(v)
fetched = time.Now()
hasVal = true
return value
}
}
// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling
// of the transcode dir so it lives alongside other derived media.
func ebookConversionCacheDir(cfg *config.Config) string {
base := os.TempDir()
if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" {
base = filepath.Dir(cfg.Playback.TranscodeDir)
}
return filepath.Join(base, "silo-ebook-epub")
}
func isTruthySetting(v string) bool {
switch strings.ToLower(strings.TrimSpace(v)) {
case "true", "1", "yes", "on", "enabled":
return true
default:
return false
}
}