* Kindle->EPUB conversion: design + proven wasm build pipeline
Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.
This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
9 review fixes folded in: failure contract, strong cache key + negative
cache, wazero command-module specifics, FS-sandbox tightening,
double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
built on amd64. go:embed target for the converter package (next).
Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.
Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: converter core + cache (Codex-reviewed)
internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.
18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).
Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
FS isolation ultimately relies on running as a non-root user (memory-safety
is the WASM boundary). validateEpub now requires STORED mimetype + verifies
the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint
Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.
- handlers.EbookConversion (converter + per-request flag predicate) on the read
handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
served_format, header contract}; enabled only when flag on AND converter
wired (double gate) so the Android client can decide whether to flip
mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
if it fails), cache dir is a sibling of TranscodeDir, flag read per request.
Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification
Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.
Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
the per-call timeout as transient and propagate a caller's cancel/deadline
verbatim instead of reclassifying it as a conversion failure. remember() now
only caches deterministic verdicts (DRM / failed), so a one-off timeout under
load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
context.WithoutCancel), so one caller cancelling no longer aborts the shared
work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
not FIFO.
Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
a negatively-cached source serves the failed contract, a miss advertises the
converted representation cheaply without triggering a (minute-long, ~1 GiB)
conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
capability endpoint no longer hit the DB per request.
Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
+ provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).
Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: remove ebookconvert workflow
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
248 lines
8.0 KiB
Go
248 lines
8.0 KiB
Go
package ebookconvert
|
|
|
|
import (
|
|
"archive/zip"
|
|
"context"
|
|
"errors"
|
|
"io"
|
|
"os"
|
|
"path/filepath"
|
|
"strings"
|
|
"testing"
|
|
"time"
|
|
)
|
|
|
|
func newTestConverter(t *testing.T) *Converter {
|
|
t.Helper()
|
|
c, err := NewConverter(context.Background(), Options{})
|
|
if err != nil {
|
|
t.Fatalf("NewConverter: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
|
return c
|
|
}
|
|
|
|
func TestConvert_DRMFreeProducesValidEpub(t *testing.T) {
|
|
for _, fixture := range []string{"sample-ncx.mobi", "sample-cp1252.mobi"} {
|
|
t.Run(fixture, func(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(context.Background(), filepath.Join("testdata", fixture), dst); err != nil {
|
|
t.Fatalf("Convert: %v", err)
|
|
}
|
|
// Independently re-validate the delivered file is a real EPUB.
|
|
assertValidEpub(t, dst)
|
|
})
|
|
}
|
|
}
|
|
|
|
func TestConvert_DRMProtectedRejected(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
err := c.Convert(context.Background(), filepath.Join("testdata", "sample-drm-v1.mobi"), dst)
|
|
if !errors.Is(err, ErrDRMProtected) {
|
|
t.Fatalf("got %v, want ErrDRMProtected", err)
|
|
}
|
|
if _, statErr := os.Stat(dst); !os.IsNotExist(statErr) {
|
|
t.Fatalf("DRM source must not produce an output file, but %s exists", dst)
|
|
}
|
|
}
|
|
|
|
func TestConvert_OversizeRejected(t *testing.T) {
|
|
c, err := NewConverter(context.Background(), Options{MaxSourceBytes: 1024})
|
|
if err != nil {
|
|
t.Fatalf("NewConverter: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrSourceTooLarge) {
|
|
t.Fatalf("got %v, want ErrSourceTooLarge", err)
|
|
}
|
|
}
|
|
|
|
func TestConvert_CorruptSourceFails(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
bad := filepath.Join(t.TempDir(), "bad.mobi")
|
|
if err := os.WriteFile(bad, []byte("not a mobi file at all"), 0o644); err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(context.Background(), bad, dst); !errors.Is(err, ErrConversionFailed) {
|
|
t.Fatalf("got %v, want ErrConversionFailed", err)
|
|
}
|
|
}
|
|
|
|
func TestConvert_MissingSourceFails(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(context.Background(), "testdata/does-not-exist.mobi", dst); !errors.Is(err, ErrConversionFailed) {
|
|
t.Fatalf("got %v, want ErrConversionFailed", err)
|
|
}
|
|
}
|
|
|
|
func TestConvert_Concurrent(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
const n = 6
|
|
errs := make(chan error, n)
|
|
for i := 0; i < n; i++ {
|
|
go func(i int) {
|
|
dst := filepath.Join(t.TempDir(), "c.epub")
|
|
errs <- c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
|
|
}(i)
|
|
}
|
|
for i := 0; i < n; i++ {
|
|
if err := <-errs; err != nil {
|
|
t.Fatalf("concurrent Convert: %v", err)
|
|
}
|
|
}
|
|
}
|
|
|
|
func TestConvert_ContextCancelled(t *testing.T) {
|
|
c := newTestConverter(t)
|
|
ctx, cancel := context.WithCancel(context.Background())
|
|
cancel() // already canceled before the conversion starts
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(ctx, filepath.Join("testdata", "sample-ncx.mobi"), dst); err == nil {
|
|
t.Fatal("expected error on canceled context")
|
|
}
|
|
}
|
|
|
|
func TestValidateEpub_RejectsNonEpubZip(t *testing.T) {
|
|
// A zip that is not an EPUB (no mimetype-first) must be rejected.
|
|
p := filepath.Join(t.TempDir(), "plain.zip")
|
|
f, err := os.Create(p)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
zw := zip.NewWriter(f)
|
|
w, _ := zw.Create("hello.txt")
|
|
_, _ = w.Write([]byte("hi"))
|
|
_ = zw.Close()
|
|
_ = f.Close()
|
|
if err := validateEpub(p); err == nil {
|
|
t.Fatal("expected validateEpub to reject a non-EPUB zip")
|
|
}
|
|
}
|
|
|
|
func TestValidateEpub_RejectsDeflatedMimetype(t *testing.T) {
|
|
// mimetype present + correct content but DEFLATED (not stored) -> reject.
|
|
p := filepath.Join(t.TempDir(), "bad.epub")
|
|
f, err := os.Create(p)
|
|
if err != nil {
|
|
t.Fatal(err)
|
|
}
|
|
zw := zip.NewWriter(f)
|
|
w, _ := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Deflate})
|
|
_, _ = w.Write([]byte("application/epub+zip"))
|
|
_ = zw.Close()
|
|
_ = f.Close()
|
|
if err := validateEpub(p); err == nil {
|
|
t.Fatal("expected rejection of deflated mimetype")
|
|
}
|
|
}
|
|
|
|
func TestConvert_AfterCloseReturnsUnavailable(t *testing.T) {
|
|
c, err := NewConverter(context.Background(), Options{})
|
|
if err != nil {
|
|
t.Fatalf("NewConverter: %v", err)
|
|
}
|
|
_ = c.Close(context.Background())
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrUnavailable) {
|
|
t.Fatalf("got %v, want ErrUnavailable", err)
|
|
}
|
|
}
|
|
|
|
func TestConvert_TimeoutClassifiedNotRawExit(t *testing.T) {
|
|
// A 1ns timeout must surface as ErrConversionTimedOut (a transient verdict,
|
|
// distinct from a deterministic failure), never as a bogus "mobitool exit
|
|
// <huge>" code.
|
|
c, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond})
|
|
if err != nil {
|
|
t.Fatalf("NewConverter: %v", err)
|
|
}
|
|
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
|
dst := filepath.Join(t.TempDir(), "out.epub")
|
|
err = c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
|
|
if !errors.Is(err, ErrConversionTimedOut) {
|
|
t.Fatalf("got %v, want ErrConversionTimedOut", err)
|
|
}
|
|
if errors.Is(err, ErrConversionFailed) {
|
|
t.Fatalf("a timeout must not also satisfy ErrConversionFailed (it would be negatively cached): %v", err)
|
|
}
|
|
if strings.Contains(err.Error(), "exit ") {
|
|
t.Fatalf("timeout leaked a raw exit code: %v", err)
|
|
}
|
|
}
|
|
|
|
// classifyRunError must propagate a caller's deadline as context.DeadlineExceeded
|
|
// (CR review): reclassifying it as a conversion failure breaks upstream
|
|
// cancellation handling and would poison the negative cache.
|
|
func TestClassifyRunError_ParentDeadlinePropagates(t *testing.T) {
|
|
parent, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
|
|
defer cancel()
|
|
run := parent // run derives from parent, so it is also past-deadline
|
|
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
|
|
if !errors.Is(err, context.DeadlineExceeded) {
|
|
t.Fatalf("got %v, want context.DeadlineExceeded propagated", err)
|
|
}
|
|
if errors.Is(err, ErrConversionFailed) || errors.Is(err, ErrConversionTimedOut) {
|
|
t.Fatalf("a caller deadline must not be reclassified as a conversion verdict: %v", err)
|
|
}
|
|
}
|
|
|
|
// When the caller's context is healthy but our own per-call timeout fired, the
|
|
// result is the transient ErrConversionTimedOut sentinel (not ErrConversionFailed).
|
|
func TestClassifyRunError_OwnTimeoutIsTimedOutSentinel(t *testing.T) {
|
|
parent := context.Background() // caller healthy
|
|
run, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
|
|
defer cancel()
|
|
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
|
|
if !errors.Is(err, ErrConversionTimedOut) {
|
|
t.Fatalf("got %v, want ErrConversionTimedOut", err)
|
|
}
|
|
if errors.Is(err, ErrConversionFailed) {
|
|
t.Fatalf("own timeout must not satisfy ErrConversionFailed: %v", err)
|
|
}
|
|
}
|
|
|
|
func assertValidEpub(t *testing.T, path string) {
|
|
t.Helper()
|
|
zr, err := zip.OpenReader(path)
|
|
if err != nil {
|
|
t.Fatalf("open epub: %v", err)
|
|
}
|
|
defer zr.Close()
|
|
if len(zr.File) == 0 || zr.File[0].Name != "mimetype" {
|
|
t.Fatalf("first entry not mimetype")
|
|
}
|
|
rc, _ := zr.File[0].Open()
|
|
mt, _ := io.ReadAll(rc)
|
|
rc.Close()
|
|
if strings.TrimSpace(string(mt)) != "application/epub+zip" {
|
|
t.Fatalf("bad mimetype %q", mt)
|
|
}
|
|
var hasContainer, hasContent bool
|
|
for _, fl := range zr.File {
|
|
if fl.Name == "META-INF/container.xml" {
|
|
hasContainer = true
|
|
}
|
|
if strings.HasPrefix(fl.Name, "OEBPS/") {
|
|
hasContent = true
|
|
}
|
|
}
|
|
if !hasContainer {
|
|
t.Fatal("missing META-INF/container.xml")
|
|
}
|
|
if !hasContent {
|
|
t.Fatal("no OEBPS content")
|
|
}
|
|
}
|
|
|
|
// Guard so the default timeout constant stays sane if edited.
|
|
func TestDefaults(t *testing.T) {
|
|
if DefaultTimeout < time.Second {
|
|
t.Fatal("DefaultTimeout too small")
|
|
}
|
|
}
|