haeaeae/silo-server

Fork 0

Files

T

History

e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171 )

* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>

2026-06-17 13:09:55 -04:00

Dockerfile

Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171 )

2026-06-17 13:09:55 -04:00

README.md

Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171 )

2026-06-17 13:09:55 -04:00

README.md

mobitool.wasm — Kindle→EPUB converter module

mobitool.wasm is libmobi's mobitool compiled to wasm32-wasi. The server runs it in-process via wazero (pure Go) to convert MOBI/AZW/AZW3 ebooks to EPUB on demand. See the design doc: docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md.

Why this shape

No cgo, no external binary. The .wasm is go:embed-ed; the Go build stays CGO_ENABLED=0 and cross-compiles trivially.
Architecture-independent. wasm32 bytecode runs identically on amd64/arm64 servers through wazero. Build once (CI, amd64), run anywhere.
Sandboxed. Untrusted ebook bytes are parsed inside the WASM sandbox with only a per-conversion scratch dir mounted — a libmobi memory bug on a malicious file cannot reach the host.

Pinned versions

Component	Version / commit
wasi-sdk	25.0
libmobi	`906274205c11944b628da1c553b255acb1af7c55`
zlib	1.3.1 (compiled to wasm)
wasmtime (smoke)	27.0.0 (build-time smoke test only)

License: libmobi is LGPL-3.0-or-later; shipping the unmodified compiled artifact + this build recipe satisfies the relink obligation. zlib is zlib-license.

Build config notes (validated by spike, 2026-06-17 on linux/amd64)

--with-libxml2=no → libmobi's internal xmlwriter provides OPF/EPUB output, so the -e (create EPUB) path works with no libxml2 dependency.
libmobi is built against a wasm-compiled zlib (not --with-zlib=no). Forcing --with-zlib=no makes libmobi vendor its own miniz, which then collides with mobitool's separate miniz (used for EPUB zip creation) — wasm-ld rejects the duplicate symbols. Real zlib for the library + the tool's miniz for zip = no clash.

Rebuild + install the artifact

docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm
id=$(docker create mobitool-wasm)
docker cp "$id:/mobitool.wasm"        internal/ebookconvert/mobitool.wasm
docker cp "$id:/mobitool.wasm.sha256" internal/ebookconvert/mobitool.wasm.sha256
docker rm "$id"

The build runs a smoke conversion of a bundled libmobi sample and fails if no EPUB is produced. mobitool -v embeds __DATE__/__TIME__, so rebuilds are not byte-for-byte reproducible — rely on the smoke test + recorded sha256, not a bit-identical rebuild check.

Runtime behavior characterized by the spike

Converts MOBI6, KF8/AZW3 hybrids, HUFF/CDIC-compressed, and unicode samples to well-formed EPUB (mimetype first = application/epub+zip, META-INF/container.xml, OEBPS/*).
DRM: mobitool prints Document is encrypted to stdout but still exits 0 and writes an EPUB (garbage content when it lacks the key). So the converter must capture stdout and validate the output is a usable EPUB — exit code alone is not a reliable DRM/failure signal.
- ⚠️ Brittleness note: DRM/print-replica detection matches the English stdout strings mobitool emits (drmMarkers / printReplicaMarker in internal/ebookconvert/converter.go). When bumping the pinned libmobi commit, re-check those messages still match — a wording change silently downgrades a DRM book to a generic "no EPUB produced" failure (it still falls back to the raw original safely, just without the explicit DRM signal).