diff --git a/docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md b/docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md new file mode 100644 index 00000000..7b14ee96 --- /dev/null +++ b/docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md @@ -0,0 +1,96 @@ +# Server-side Kindle→EPUB conversion (MOBI / AZW / AZW3) — design + +**Goal:** Let the silo-server render Kindle-family ebooks (mobi, azw, azw3) in the Android in-app reader by converting them to EPUB **server-side, in-process, on demand**, served transparently through the existing read endpoint. No external converter binary, no cgo. + +**Status:** Approach approved by the user: convert via **libmobi**, run **built-in** to the server as **WASM executed by `wazero`** (a pure-Go runtime). On-demand + cached, admin-gated, DRM-free only. This is the MOBI half of the "all 9 server formats in-app" goal; the CBR (client-side) half is a separate Android spec. + +> **Linchpin PROVEN (build spike, 2026-06-17, linux/amd64).** libmobi + `mobitool` cross-compile to `wasm32-wasi` (wasi-sdk 25, libmobi `9062742`, zlib 1.3.1 built to wasm); `-e` (create EPUB) is present; the module converts real MOBI6 / KF8-hybrid / HUFF-CDIC / unicode samples to **well-formed EPUB** (`mimetype`=`application/epub+zip` first, `META-INF/container.xml`, `OEBPS/*`). Verified end-to-end under **wazero** (the prod runtime) with WASI preopen + argv + `_start` + exit handling. Reproducible build stage committed at `tools/mobitool-wasm/Dockerfile` (with a smoke-conversion gate). Build-config gotcha resolved: build libmobi against a **wasm-compiled zlib** — `--with-zlib=no` makes libmobi vendor miniz that collides with mobitool's own miniz at `wasm-ld`. **DRM finding:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0** and writes a (garbage) EPUB — so detect failure by capturing stdout + validating output, never by exit code alone. + +## Why this shape + +- Pure-Go MOBI/KF8 *parsers* don't exist at usable quality (the Go MOBI packages only *write*). Calibre is the gold standard but ~300MB+ of Python/Qt — rejected as too heavy. **libmobi** (mature C library behind the `mobitool` CLI) handles MOBI6 + KF8/AZW3 + AZW, decompression, and EPUB generation. +- Rather than shell out to an external `mobitool` binary (a deploy moving part) or bind via cgo (forces `CGO_ENABLED=1`, a C toolchain, painful cross-compile, and an LGPLv3 static-link obligation), we compile **`mobitool` to a WASI `.wasm`** once and **run it in-process with `wazero`**. +- This keeps the main Go build **`CGO_ENABLED=0`** and pure; one **architecture-independent** `.wasm` runs on amd64/arm64 servers alike; the `.wasm` is `go:embed`-ed into the binary (self-contained); libmobi's LGPL is cleanly isolated as a separate embedded artifact; and — a real bonus — **untrusted ebook bytes are parsed inside the WASM sandbox**, so a memory-safety bug in libmobi on a malicious file cannot reach the host (only the mapped temp dir). + +## Components + +### 1. The WASM module (build-time artifact) +- Compile libmobi + its `mobitool` tool to `wasm32-wasi` using the **wasi-sdk** (clang). Output: `mobitool.wasm`. +- Build libmobi **without libxml2** (`--with-libxml2=no` / `--disable-xmlwriter` as available) so the module has no external native deps; mobitool's EPUB writer uses a bundled miniz, which is portable C and compiles to WASI cleanly. (Feasibility of the no-libxml2 build for the `-e` path is the #1 thing to verify in the build spike — see Risks.) +- Build it in a **dedicated Docker build stage** (or a committed `Makefile`/script under `tools/mobitool-wasm/`). **Commit the resulting `mobitool.wasm`** into the repo so the normal Go build does not need the wasi-sdk — it only `go:embed`s the prebuilt artifact. Rebuild the `.wasm` only when bumping libmobi; record the libmobi version + build flags next to it. + +### 2. `internal/ebookconvert` (new package) — the in-process converter +```go +// Converter converts a Kindle-family ebook file to EPUB bytes, in-process via wazero. +type Converter interface { + // Convert reads srcPath (mobi/azw/azw3) and returns the path to a converted .epub. + // Returns ErrDRMProtected / ErrUnsupported / ErrConversionFailed as appropriate. + Convert(ctx context.Context, srcPath string) (epubPath string, err error) +} +``` +- `//go:embed mobitool.wasm` → compile the module **once** at startup with `wazero.NewRuntime` + `CompileModule` (the compiled module is cached and reused; only cheap instantiation happens per conversion). +- **wazero command-module specifics** (from Codex review): `wasi_snapshot_preview1.MustInstantiate` the host module once; `Runtime.CompileModule` once at startup; per conversion instantiate with `ModuleConfig.WithArgs(...)` (argv ≈ `mobitool -e -o `), `WithFSConfig`, `WithStdout`+`WithStderr` (capture **both** — `mobitool.c` prints many failures with `printf` to stdout, not stderr), and the default `_start`. Use `WithName("")` (or a unique name) for repeated instantiation. Non-zero exits surface as `*sys.ExitError`. Honor ctx timeout via `RuntimeConfig.WithCloseOnContextDone(true)`. +- **Filesystem sandboxing** (from Codex review): wazero's `WithDirMount`/`WithReadOnlyDirMount` still permit relative `../../` traversal *within host permissions*, so **never preopen the library directory or the cache root**. Per conversion: copy the source into a private scratch tree, mount input **read-only via an `fs.FS`** (`WithFSMount(os.DirFS(in), …)` — confined, no `..` escape) and a dedicated empty **writable** output dir (`WithDirMount`). **The WASM sandbox is a memory-safety boundary, not a hard filesystem jail** — a writable dir mount can still be traversed by a compromised guest within the server user's reach. Filesystem isolation therefore depends on a **deployment requirement: run silo-server as a constrained, non-root service user (the container already does), so the converter's blast radius is minimal.** This is documented here and in `internal/ebookconvert/converter.go`. +- Enforce a wall-clock **timeout** (ctx), a **max source size**, and a **concurrency semaphore** (N simultaneous conversions). WASM memory is bounded by wazero config. **Resource note:** the defaults are `DefaultMaxMemoryPages` = 1 GiB linear memory per conversion × `DefaultConcurrency` = 2, i.e. up to ~2 GiB of wasm memory under saturation (plus the 256 MiB source / 512 MiB output caps). Generous for a small server; tune `ebookconvert.Options` for tight deployments. +- Map mobitool's exit code / captured stdout+stderr to typed errors. **Deterministic** verdicts (cacheable): DRM-protected input → `ErrDRMProtected`; corrupt / unconvertible layout / oversize-output → `ErrConversionFailed`. **Transient** verdicts (never negatively cached, retried next read): the per-call timeout firing → `ErrConversionTimedOut`; source over the size cap → `ErrSourceTooLarge`; caller cancellation/deadline → the context error propagated verbatim. + +### 3. Conversion cache (on-demand, keyed) +- Cache converted EPUBs on disk under a configured cache root, keyed by **strong source identity + converter identity** — `file_id + size + mtime_ns + content hash` (use the scanner checksum if one exists) **plus** the converter **module fingerprint** (`moduleVersion`: a sha256 of the embedded `mobitool.wasm`, which already pins the libmobi commit, build flags, and wasi-sdk used to produce it). (Plain `path|size|mtime` can false-hit when a file is replaced with preserved mtime/size — Codex.) A stale key misses and reconverts. +- **Singleflight** per key so concurrent reads of the same just-opened book convert once. The shared conversion runs on a **detached context** (decoupled from any one caller), so a single caller cancelling its request never aborts the work for the others. +- **Negative caching**: remember only **deterministic-bad** sources — DRM (`ErrDRMProtected`) and conversion failures (`ErrConversionFailed`) — so every reader open does not reconvert a known-bad file. Transient outcomes (timeout, cancellation, oversize source) are **not** cached so they retry. +- A cache **hit refreshes the entry's mtime**, so the mtime-ordered budget eviction acts as a real LRU; eviction never deletes the entry it is handing back or another conversion's in-flight temp. +- Bounded by an LRU / total-size budget with opportunistic eviction (mirrors the existing reader-file/image cache conventions in the codebase). + +### 4. Wiring into the read path +- `serveEbookInline` (`internal/api/handlers/ebook_reader.go:602`) currently `os.Open`s `file.FilePath` and `http.ServeContent`s it with `ebookMimeType(...)`. +- Change: when the source format is **kindle-family AND conversion is enabled**, resolve the converted EPUB via the cache (convert-on-miss), then serve **that** file with `Content-Type: application/epub+zip`. All other formats are served exactly as today (raw, byte-range, unchanged). +- Byte-range/`http.ServeContent` semantics are preserved — we serve the cached EPUB `*os.File` the same way, so resume/range still works. +- **Response headers** (from Codex review): `Content-Type: application/epub+zip`, `Content-Disposition` with an `.epub` filename, a strong `ETag` derived from the conversion cache key, and — because the *same URL* can return raw or converted bytes depending on the setting/failure — `Cache-Control: private, max-age=0, must-revalidate` (or `no-store`) to prevent stale client/proxy caching of the wrong representation. +- **HEAD is cache-only.** A `HEAD /read` must never trigger a (possibly minute-long, ~1 GiB) conversion. It answers from the cache: a hit serves the real converted headers + `Content-Length`; a negatively-cached source serves the `failed` contract; a miss advertises the converted headers (`Content-Type`, the cache-key `ETag`, `X-Silo-Ebook-Conversion: converted`) **optimistically with no body and no conversion** — the subsequent `GET` produces the body and the authoritative DRM/failure verdict, which then populates the cache that later HEADs read. *(Client note: a HEAD miss intentionally omits `Content-Length`; coordinate with the Android/Apple clients if any relies on it.)* + +### 5. Admin gate + client capability +- An **admin-toggleable setting** (`ebook.kindle_conversion_enabled`, default off) gates the whole feature, via the existing server settings mechanism (`internal/api/handlers/settings.go` / `admin.go` `HandleUpdateSetting`). The flag is read with a short TTL cache so the hot read path (and the capability endpoint clients poll) does not hit the DB per request, while toggles still apply within seconds without a restart. When off: kindle-family files serve raw as today and the capability is not advertised. +- **Two gates to advertise the capability** (from Codex review): advertise conversion availability to clients **only when** (a) the admin flag is on **and** (b) the embedded WASM module compiled + instantiated successfully at startup. The Android client flips mobi/azw/azw3 from `ExternalOnly`→`InApp` **only when the flag is advertised**; otherwise it keeps today's external-open fallback. (Android-side change is small and tracked separately; this spec is server-side.) +- **Capability endpoint (implemented):** `GET /api/v1/ebooks/capability` → `{enabled, source_formats:["mobi","azw","azw3"], served_format:"epub", header:"X-Silo-Ebook-Conversion", header_failed_value:"failed"}`. `enabled` is true only when both gates hold (`EbookConversion.active`). The read response carries `X-Silo-Ebook-Conversion: converted|failed` so the client knows whether the body is EPUB or the raw original (→ external-open). ETag on the converted response is derived from the **exact conversion cache key** (`SourceKey.CacheKey()`) so a validator can never disagree with the cache; the failed/raw fallback is `Cache-Control: no-store` since the same URL flips representations with the flag. + +## Data flow (read of a `.azw3`, conversion enabled) +1. `GET /api/v1/ebooks/{cid}/files/{fid}/read` → authorize → `file` is `azw3`, conversion enabled. +2. Cache lookup by source identity+version. Hit → serve cached `.epub`. Miss → singleflight: `Converter.Convert` (wazero runs `mobitool.wasm` on a temp copy) → store under key. +3. `http.ServeContent` the `.epub` with `application/epub+zip`, byte-range supported. +4. Client sees EPUB → existing reflow `EpubReader` renders it. Reader progress stays keyed to the **source** `fileId` (the EPUB is a derived view of the same file), so progress/annotations are unaffected. + +## Error handling — explicit client failure contract +Transparent raw fallback is **not** safe on its own: once the server advertises conversion and the Android in-app EPUB path opens `/read`, returning raw MOBI/AZW bytes hands the EPUB reader a non-EPUB body (Codex). So failures need a contract the client can detect: +- On conversion failure (DRM, corrupt, timeout, oversize) the read endpoint returns a **structured signal** the Android client maps to its external-open fallback — preferred: a distinct status (e.g. `422 Unprocessable` / `415`) with an error code, **or** serve raw bytes **plus** an explicit `X-Silo-Ebook-Conversion: failed` (with the original MIME) header that the client checks. Pick one and make both sides honor it. Never silently hand raw bytes to the EPUB path. +- `ErrDRMProtected` vs `ErrConversionFailed` are distinguished in logs + (optionally) the error code, so DRM books get the "open externally" message and transient failures can be retried. +- Module compile/instantiate failure at startup (corrupt embed) → capability **not advertised** (gate b), kindle files serve raw as today, logged loudly. + +## Testing +- `internal/ebookconvert` unit tests with **small DRM-free MOBI6, AZW, and KF8/AZW3 fixtures**: `Convert` produces a valid EPUB (zip with `mimetype` first + `META-INF/container.xml` + OPF + at least one content doc), deterministic across runs; corrupt input → `ErrConversionFailed`; oversize → rejected before invocation; timeout honored (ctx cancel). +- Cache tests: miss→convert→hit; key changes on source mtime/size and on module-version bump; singleflight collapses concurrent conversions. +- Handler tests: enabled+kindle → `application/epub+zip` body is a valid EPUB; disabled → raw bytes + original MIME; non-kindle → unchanged; DRM/failed → graceful fallback path. +- A build-stage check (CI) that the committed `mobitool.wasm` matches a rebuild from the pinned libmobi (or at least that it instantiates + converts a fixture), so the artifact can't silently rot. + +## Sequencing — spike first (Codex-recommended) +**The first PR is only the build spike + a tiny Go harness that converts fixtures** — no handler/cache/wiring until the linchpin is proven. Codex confirmed libmobi documents both deps as optional (`--with-zlib=no` → bundled `miniz.c`; `--with-libxml2=no` → internal xmlwriter) and that `mobitool.c` gates `-e` behind `USE_XMLWRITER`, not `HAVE_LIBXML2` — so the no-libxml2 wasm build is *plausible*, but the spike must **assert** `mobitool -h` exposes `-e` and a fixture actually converts. Only then design the handler/cache/wiring PR. + +## Risks / spike-first items +1. **libmobi+mobitool compiles to `wasm32-wasi` with the `-e` (EPUB) path working without libxml2.** The linchpin. If libxml2 turns out mandatory for `-e`, fall back to compiling libxml2 to wasm too (heavier) or assembling the EPUB in Go from libmobi's `MOBIRawml` parts. +2. **wazero WASI preopen file IO** for the file sizes involved (tens of MB) — confirm mapping + memory/time limits. +3. mobitool CLI surface as a wasm **command module** (argv + preopens) — confirm wazero runs it via `_start` and we capture exit status + stdout + stderr. + +## Committed `mobitool.wasm` — guardrails (Codex) +Commit the artifact (so normal Go builds need no wasi-sdk) **with**: the pinned libmobi source commit, the Dockerfile/build script that produced it, its SHA256, the build flags, and the LGPL license notice. Add a **CI smoke test** that instantiates the embedded module and converts a fixture. Do not gate CI on byte-for-byte rebuild reproducibility — `mobitool -v` embeds `__DATE__`/`__TIME__`, so a rebuild won't match byte-for-byte unless normalized; rely on the fixture-conversion smoke + metadata hash instead. + +## Process gate (Codex) +`docs/architecture/v1-scope.md` indicates v1 scope is not locked and new user-facing capabilities may need a proposal / project-tracking entry **before** a feature PR. Confirm this process step with the maintainer before opening the conversion PR. *(Decision deferred to the user.)* + +## Out of scope +- CBR (separate Android client-side spec). +- DRM removal (never). +- Converting at scan/ingest time (on-demand only). +- Surfacing a separate "EPUB version" in the catalog (we serve transparently through `/read`). +- Reusing the WASM converter for other formats (later, if useful). + +## Commit / deploy rules (project) +- silo-server changes go on a **feature branch, committed locally, PR-ready — NOT pushed** (per standing rule; server changes land via PR). diff --git a/go.mod b/go.mod index 6a6e3397..2012f0ca 100644 --- a/go.mod +++ b/go.mod @@ -31,6 +31,7 @@ require ( github.com/oklog/ulid/v2 v2.1.0 github.com/pgvector/pgvector-go v0.3.0 github.com/pressly/goose/v3 v3.27.1 + github.com/tetratelabs/wazero v1.12.0 github.com/wneessen/go-mail v0.7.3 github.com/zishang520/socket.io/v2 v2.5.0 go.n16f.net/thumbhash v1.1.0 diff --git a/go.sum b/go.sum index 369ce3df..f9f7069c 100644 --- a/go.sum +++ b/go.sum @@ -197,6 +197,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/ github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU= +github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0= github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo= github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs= github.com/uptrace/bun v1.1.12 h1:sOjDVHxNTuM6dNGaba0wUuz7KvDE1BmNu9Gqs2gJSXQ= diff --git a/internal/api/ebook_conversion.go b/internal/api/ebook_conversion.go new file mode 100644 index 00000000..57bad66c --- /dev/null +++ b/internal/api/ebook_conversion.go @@ -0,0 +1,103 @@ +package api + +import ( + "context" + "log/slog" + "os" + "path/filepath" + "strings" + "sync" + "time" + + "github.com/Silo-Server/silo-server/internal/api/handlers" + "github.com/Silo-Server/silo-server/internal/catalog" + "github.com/Silo-Server/silo-server/internal/config" + "github.com/Silo-Server/silo-server/internal/ebookconvert" +) + +// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB +// conversion. Off by default; toggled via PUT /admin/settings/{key}. +const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled" + +// buildEbookConversion wires the in-process Kindle->EPUB converter for the read +// handler. It compiles the embedded WASM module once at startup; if that fails +// the feature stays off (returns nil) and the raw original is served as before. +// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be +// toggled without a restart yet does not hit the DB on every read. +// +// The Converter (a wazero runtime, ~no external resources — per-conversion +// scratch dirs are cleaned as they go) is intentionally owned for the process +// lifetime: it is not Close()d on shutdown because process exit reclaims it and +// there is nothing to flush. Memory is bounded per conversion by +// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's +// resource notes); tune via ebookconvert.Options if the deployment is tight. +func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion { + if settings == nil { + return nil + } + converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{}) + if err != nil { + slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err) + return nil + } + cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())}) + if err != nil { + slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err) + _ = converter.Close(context.Background()) + return nil + } + slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey) + return &handlers.EbookConversion{ + Converter: cache, + Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey), + } +} + +// ebookFlagPredicate returns a cached boolean reader for a server setting. The +// read path (every ebook read + the capability endpoint, which clients may +// poll) would otherwise issue a DB query per request; a short TTL collapses +// bursts to one query while keeping admin toggles effectively instant. On a read +// error it keeps the last known value rather than flapping the feature off. +func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool { + const ttl = 3 * time.Second + var ( + mu sync.Mutex + value bool + fetched time.Time + hasVal bool + ) + return func(ctx context.Context) bool { + mu.Lock() + defer mu.Unlock() + if hasVal && time.Since(fetched) < ttl { + return value + } + v, err := settings.Get(ctx, key) + if err != nil { + return value // keep last known (false until the first successful read) + } + value = isTruthySetting(v) + fetched = time.Now() + hasVal = true + return value + } +} + +// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling +// of the transcode dir so it lives alongside other derived media. +func ebookConversionCacheDir(cfg *config.Config) string { + base := os.TempDir() + if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" { + base = filepath.Dir(cfg.Playback.TranscodeDir) + } + return filepath.Join(base, "silo-ebook-epub") +} + +func isTruthySetting(v string) bool { + switch strings.ToLower(strings.TrimSpace(v)) { + case "true", "1", "yes", "on", "enabled": + return true + default: + return false + } +} diff --git a/internal/api/ebook_conversion_test.go b/internal/api/ebook_conversion_test.go new file mode 100644 index 00000000..317392be --- /dev/null +++ b/internal/api/ebook_conversion_test.go @@ -0,0 +1,39 @@ +package api + +import ( + "context" + "sync/atomic" + "testing" +) + +// stubSettings counts Get calls so we can assert the flag predicate caches. +type stubSettings struct { + value string + gets int64 +} + +func (s *stubSettings) Get(_ context.Context, _ string) (string, error) { + atomic.AddInt64(&s.gets, 1) + return s.value, nil +} +func (s *stubSettings) Set(context.Context, string, string) error { return nil } +func (s *stubSettings) GetAll(context.Context) (map[string]string, error) { + return map[string]string{}, nil +} + +// The flag predicate must not hit the settings store on every call: the read +// path (every ebook read + capability poll) would otherwise issue a DB query +// per request. +func TestEbookFlagPredicate_CachesWithinTTL(t *testing.T) { + s := &stubSettings{value: "true"} + pred := ebookFlagPredicate(s, "ebook.kindle_conversion_enabled") + + for i := 0; i < 50; i++ { + if !pred(context.Background()) { + t.Fatalf("call %d: predicate returned false for a truthy flag", i) + } + } + if got := atomic.LoadInt64(&s.gets); got > 2 { + t.Fatalf("settings.Get called %d times across 50 predicate calls; expected it to be cached", got) + } +} diff --git a/internal/api/handlers/ebook_convert_serve.go b/internal/api/handlers/ebook_convert_serve.go new file mode 100644 index 00000000..6bff9da7 --- /dev/null +++ b/internal/api/handlers/ebook_convert_serve.go @@ -0,0 +1,184 @@ +package handlers + +import ( + "context" + "errors" + "fmt" + "net/http" + "os" + "path/filepath" + "strings" + + "github.com/Silo-Server/silo-server/internal/ebookconvert" + "github.com/Silo-Server/silo-server/internal/models" +) + +// ConversionHeader tells the client how a kindle-family read was served. +// "converted" => body is EPUB; "failed" => body is the raw original and the +// client should fall back to opening it externally. Absent => no conversion +// was attempted (feature off or non-kindle format), serve/behave as before. +const ConversionHeader = "X-Silo-Ebook-Conversion" + +// EbookConverter produces (and caches) a converted EPUB for a source file. +// Implemented by *ebookconvert.Cache; an interface here for testability. +type EbookConverter interface { + GetOrConvert(ctx context.Context, srcPath string, key ebookconvert.SourceKey) (string, error) + // Lookup reports a cached conversion WITHOUT converting (used by HEAD so a + // metadata probe never kicks off a full conversion): a path on hit, a + // non-nil error for a negatively-cached source, else a miss. + Lookup(key ebookconvert.SourceKey) (string, error, bool) +} + +// EbookConversion bundles the converter with the admin-flag predicate. A nil +// *EbookConversion (or nil Converter) means the feature is off. +type EbookConversion struct { + Converter EbookConverter + // Enabled reports whether the admin flag is on for this request. + Enabled func(ctx context.Context) bool +} + +func (c *EbookConversion) active(ctx context.Context) bool { + return c != nil && c.Converter != nil && c.Enabled != nil && c.Enabled(ctx) +} + +// kindleReaderFormats are the Kindle-family formats the converter handles. +var kindleReaderFormats = map[string]bool{"mobi": true, "azw": true, "azw3": true} + +func isKindleReaderFile(file *models.MediaFile) bool { + return file != nil && kindleReaderFormats[ebookReaderFormat(file.FilePath, file.Container)] +} + +// serveEbook serves an ebook file, converting Kindle-family formats to EPUB +// when the feature is enabled. On conversion failure it serves the raw original +// with the ConversionHeader=failed contract so the client opens it externally. +func (h *EbookReaderHandler) serveEbook(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error { + if h.Conversion.active(r.Context()) && isKindleReaderFile(file) { + // Strong identity for both the conversion cache and the response ETag: + // id + size + mtime (stat) + content hash + module version. + key, statErr := ebookconvert.SourceKeyFromStat(file.ID, file.FilePath) + if statErr != nil { + return h.serveRawKindleFallback(w, r, file) + } + key.Checksum = file.FileHash + + // HEAD must not trigger a (possibly minute-long, 1 GiB) conversion. Answer + // from cache only; on a miss, advertise the converted representation + // cheaply and let the GET produce the body + authoritative verdict. + if r.Method == http.MethodHead { + return h.serveKindleHead(w, r, file, key) + } + + epubPath, err := h.Conversion.Converter.GetOrConvert(r.Context(), file.FilePath, key) + switch { + case err == nil: + if serveErr := serveConvertedEpub(w, r, file, epubPath, key); serveErr != nil { + // Produced but unservable (open/stat) — honor the contract, not a 500. + return h.serveRawKindleFallback(w, r, file) + } + return nil + case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded): + // Caller went away / request cancelled — not a conversion verdict. + return err + default: + // DRM, corrupt, oversize, timed out, unavailable: fall back to the raw + // original and tell the client so it can open externally. + return h.serveRawKindleFallback(w, r, file) + } + } + // Feature off or non-kindle format: serve the raw original with its native + // MIME and no conversion header. This path deliberately omits the failed + // contract — when conversion is disabled the capability endpoint already + // reports `enabled:false`, so clients keep their external-open path and never + // expect an EPUB here. (No representation flip to guard against.) + return serveEbookInline(w, r, file) +} + +// serveRawKindleFallback serves the raw original with the failed-conversion +// contract. no-store because the same URL flips raw/converted with the flag. +func (h *EbookReaderHandler) serveRawKindleFallback(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error { + w.Header().Set(ConversionHeader, "failed") + w.Header().Set("Cache-Control", "no-store") + return serveEbookInline(w, r, file) +} + +// serveKindleHead answers a HEAD for a kindle-family file without converting: +// - cache hit → real converted headers (and ServeContent's Content-Length); +// - neg-cached → the failed contract (raw original headers); +// - miss → converted headers advertised optimistically (no body, no +// Content-Length, no conversion). A HEAD is only a hint; the GET delivers +// the body and the authoritative DRM/failure verdict, which then populates +// the (positive or negative) cache that subsequent HEADs read. +func (h *EbookReaderHandler) serveKindleHead(w http.ResponseWriter, r *http.Request, file *models.MediaFile, key ebookconvert.SourceKey) error { + path, lookupErr, ok := h.Conversion.Converter.Lookup(key) + switch { + case ok: + if serveErr := serveConvertedEpub(w, r, file, path, key); serveErr != nil { + return h.serveRawKindleFallback(w, r, file) + } + return nil + case lookupErr != nil: + return h.serveRawKindleFallback(w, r, file) + default: + setConvertedEpubHeaders(w, file, key) + w.WriteHeader(http.StatusOK) + return nil + } +} + +// setConvertedEpubHeaders writes the headers for a converted-EPUB response. +// Because the same URL can return raw or converted bytes depending on the +// flag/outcome, the ETag is derived from the exact conversion cache key (so a +// validator can never disagree with the cache) with a revalidate policy that +// keeps clients/proxies from serving a stale representation. +func setConvertedEpubHeaders(w http.ResponseWriter, file *models.MediaFile, key ebookconvert.SourceKey) { + name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub" + w.Header().Set("Content-Type", "application/epub+zip") + w.Header().Set("Content-Disposition", inlineContentDisposition(name)) + w.Header().Set("X-Content-Type-Options", "nosniff") + w.Header().Set(ConversionHeader, "converted") + w.Header().Set("ETag", fmt.Sprintf("%q", "epubconv-"+key.CacheKey())) + w.Header().Set("Cache-Control", "private, max-age=0, must-revalidate") +} + +// serveConvertedEpub serves the cached EPUB with EPUB headers (see +// setConvertedEpubHeaders). For a HEAD, ServeContent emits the headers and +// Content-Length without a body. +func serveConvertedEpub(w http.ResponseWriter, r *http.Request, file *models.MediaFile, epubPath string, key ebookconvert.SourceKey) error { + f, err := os.Open(epubPath) + if err != nil { + return fmt.Errorf("opening converted epub: %w", err) + } + defer f.Close() + stat, err := f.Stat() + if err != nil { + return fmt.Errorf("stat converted epub: %w", err) + } + + name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub" + setConvertedEpubHeaders(w, file, key) + http.ServeContent(w, r, name, stat.ModTime(), f) + return nil +} + +// ebookConversionCapability is the client-facing advertisement for the +// Kindle->EPUB feature. Clients flip mobi/azw/azw3 to in-app reading only when +// Enabled is true; otherwise they keep the external-open path. +type ebookConversionCapability struct { + Enabled bool `json:"enabled"` + SourceFormats []string `json:"source_formats"` + ServedFormat string `json:"served_format"` + Header string `json:"header"` + HeaderOnFailed string `json:"header_failed_value"` +} + +// HandleConversionCapability reports whether Kindle->EPUB conversion is active +// (admin flag on AND the converter initialized). GET /api/v1/ebooks/capability. +func (h *EbookReaderHandler) HandleConversionCapability(w http.ResponseWriter, r *http.Request) { + writeJSON(w, http.StatusOK, ebookConversionCapability{ + Enabled: h.Conversion.active(r.Context()), + SourceFormats: []string{"mobi", "azw", "azw3"}, + ServedFormat: "epub", + Header: ConversionHeader, + HeaderOnFailed: "failed", + }) +} diff --git a/internal/api/handlers/ebook_convert_serve_test.go b/internal/api/handlers/ebook_convert_serve_test.go new file mode 100644 index 00000000..f33451c1 --- /dev/null +++ b/internal/api/handlers/ebook_convert_serve_test.go @@ -0,0 +1,271 @@ +package handlers + +import ( + "context" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/Silo-Server/silo-server/internal/ebookconvert" + "github.com/Silo-Server/silo-server/internal/models" +) + +type fakeConverter struct { + epubPath string + err error + calls int + + lookupPath string + lookupErr error + lookupOK bool +} + +func (f *fakeConverter) GetOrConvert(_ context.Context, _ string, _ ebookconvert.SourceKey) (string, error) { + f.calls++ + return f.epubPath, f.err +} + +func (f *fakeConverter) Lookup(_ ebookconvert.SourceKey) (string, error, bool) { + return f.lookupPath, f.lookupErr, f.lookupOK +} + +func writeTemp(t *testing.T, name, content string) string { + t.Helper() + p := filepath.Join(t.TempDir(), name) + if err := os.WriteFile(p, []byte(content), 0o644); err != nil { + t.Fatal(err) + } + return p +} + +func serveAndRecord(t *testing.T, h *EbookReaderHandler, file *models.MediaFile) *httptest.ResponseRecorder { + t.Helper() + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodGet, "/read", nil) + if err := h.serveEbook(rec, req, file); err != nil { + t.Fatalf("serveEbook: %v", err) + } + return rec +} + +func enabledConversion(conv EbookConverter, on bool) *EbookConversion { + return &EbookConversion{Converter: conv, Enabled: func(context.Context) bool { return on }} +} + +func TestServeEbook_ConvertsKindleWhenEnabled(t *testing.T) { + epub := writeTemp(t, "out.epub", "PK\x03\x04 fake-epub-bytes") + src := writeTemp(t, "book.azw3", "raw-azw3-source") + conv := &fakeConverter{epubPath: epub} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + + file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3", FileHash: "abc123"} + rec := serveAndRecord(t, h, file) + + if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" { + t.Fatalf("Content-Type = %q, want application/epub+zip", got) + } + if got := rec.Header().Get(ConversionHeader); got != "converted" { + t.Fatalf("%s = %q, want converted", ConversionHeader, got) + } + if rec.Header().Get("ETag") == "" { + t.Fatal("expected an ETag on converted response") + } + if rec.Body.String() != "PK\x03\x04 fake-epub-bytes" { + t.Fatalf("body = %q, want the epub bytes", rec.Body.String()) + } + if conv.calls != 1 { + t.Fatalf("converter called %d times, want 1", conv.calls) + } +} + +func TestServeEbook_ConversionFailureFallsBackToRawWithHeader(t *testing.T) { + raw := writeTemp(t, "book.mobi", "raw-mobi-bytes") + conv := &fakeConverter{err: ebookconvert.ErrDRMProtected} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + + file := &models.MediaFile{ID: 2, FilePath: raw, Container: "mobi"} + rec := serveAndRecord(t, h, file) + + if got := rec.Header().Get(ConversionHeader); got != "failed" { + t.Fatalf("%s = %q, want failed", ConversionHeader, got) + } + if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" { + t.Fatalf("Content-Type = %q, want raw mobi mime", got) + } + if rec.Body.String() != "raw-mobi-bytes" { + t.Fatalf("body = %q, want raw mobi bytes", rec.Body.String()) + } +} + +func TestServeEbook_DisabledServesRaw(t *testing.T) { + raw := writeTemp(t, "book.mobi", "raw-mobi-bytes") + conv := &fakeConverter{epubPath: "should-not-be-used"} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, false)} + + file := &models.MediaFile{ID: 3, FilePath: raw, Container: "mobi"} + rec := serveAndRecord(t, h, file) + + if got := rec.Header().Get(ConversionHeader); got != "" { + t.Fatalf("%s = %q, want empty (no conversion attempted)", ConversionHeader, got) + } + if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" { + t.Fatalf("Content-Type = %q, want raw mobi mime", got) + } + if conv.calls != 0 { + t.Fatalf("converter called %d times while disabled, want 0", conv.calls) + } +} + +func TestServeEbook_NonKindleUntouched(t *testing.T) { + raw := writeTemp(t, "book.epub", "epub-bytes") + conv := &fakeConverter{epubPath: "should-not-be-used"} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + + file := &models.MediaFile{ID: 4, FilePath: raw, Container: "epub"} + rec := serveAndRecord(t, h, file) + + if rec.Header().Get(ConversionHeader) != "" { + t.Fatal("non-kindle file must not trigger conversion") + } + if conv.calls != 0 { + t.Fatalf("converter called %d times for epub, want 0", conv.calls) + } + if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" { + t.Fatalf("Content-Type = %q, want epub", got) + } +} + +func TestServeEbook_NilConversionServesRaw(t *testing.T) { + raw := writeTemp(t, "book.mobi", "raw") + h := &EbookReaderHandler{} // Conversion nil + file := &models.MediaFile{ID: 5, FilePath: raw, Container: "mobi"} + rec := serveAndRecord(t, h, file) + if rec.Header().Get(ConversionHeader) != "" { + t.Fatal("nil Conversion must not set the conversion header") + } +} + +func TestHandleConversionCapability(t *testing.T) { + for _, tc := range []struct { + name string + conv *EbookConversion + want bool + }{ + {"enabled", enabledConversion(&fakeConverter{}, true), true}, + {"flag-off", enabledConversion(&fakeConverter{}, false), false}, + {"not-wired", nil, false}, + } { + t.Run(tc.name, func(t *testing.T) { + h := &EbookReaderHandler{Conversion: tc.conv} + rec := httptest.NewRecorder() + h.HandleConversionCapability(rec, httptest.NewRequest(http.MethodGet, "/ebooks/capability", nil)) + if rec.Code != http.StatusOK { + t.Fatalf("status = %d", rec.Code) + } + body := rec.Body.String() + wantField := `"enabled":false` + if tc.want { + wantField = `"enabled":true` + } + if !strings.Contains(body, wantField) { + t.Fatalf("body %q missing %s", body, wantField) + } + if !strings.Contains(body, `"served_format":"epub"`) || !strings.Contains(body, "mobi") { + t.Fatalf("capability body incomplete: %s", body) + } + }) + } +} + +// A HEAD on a cache miss must NOT trigger a (possibly minute-long) conversion; +// it advertises the converted representation cheaply and lets the GET do the work. +func TestServeEbook_HeadDoesNotConvertOnMiss(t *testing.T) { + src := writeTemp(t, "book.azw3", "raw-azw3-source") + conv := &fakeConverter{lookupOK: false} // cache miss + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3"} + + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodHead, "/read", nil) + if err := h.serveEbook(rec, req, file); err != nil { + t.Fatalf("serveEbook HEAD: %v", err) + } + if conv.calls != 0 { + t.Fatalf("HEAD on a cache miss triggered %d conversions, want 0", conv.calls) + } + if got := rec.Header().Get(ConversionHeader); got != "converted" { + t.Fatalf("%s = %q, want converted (advertised on HEAD miss)", ConversionHeader, got) + } + if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" { + t.Fatalf("Content-Type = %q, want application/epub+zip", got) + } + if rec.Header().Get("ETag") == "" { + t.Fatal("HEAD miss should still carry the conversion ETag") + } +} + +// A HEAD that hits the cache serves the converted headers from the cached file, +// without converting. +func TestServeEbook_HeadServesCachedConverted(t *testing.T) { + epub := writeTemp(t, "out.epub", "PK\x03\x04 cached-epub") + src := writeTemp(t, "book.mobi", "raw-mobi-source") + conv := &fakeConverter{lookupPath: epub, lookupOK: true} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + file := &models.MediaFile{ID: 2, FilePath: src, Container: "mobi"} + + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodHead, "/read", nil) + if err := h.serveEbook(rec, req, file); err != nil { + t.Fatalf("serveEbook HEAD: %v", err) + } + if conv.calls != 0 { + t.Fatalf("HEAD on a cache hit triggered %d conversions, want 0", conv.calls) + } + if got := rec.Header().Get(ConversionHeader); got != "converted" { + t.Fatalf("%s = %q, want converted", ConversionHeader, got) + } + if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" { + t.Fatalf("Content-Type = %q, want application/epub+zip", got) + } +} + +// A HEAD on a negatively-cached (DRM/failed) source reports the failed contract +// without converting. +func TestServeEbook_HeadDRMReturnsFailed(t *testing.T) { + src := writeTemp(t, "book.mobi", "raw-mobi-source") + conv := &fakeConverter{lookupErr: ebookconvert.ErrDRMProtected} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + file := &models.MediaFile{ID: 3, FilePath: src, Container: "mobi"} + + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodHead, "/read", nil) + if err := h.serveEbook(rec, req, file); err != nil { + t.Fatalf("serveEbook HEAD: %v", err) + } + if conv.calls != 0 { + t.Fatalf("HEAD on a negatively-cached source triggered %d conversions, want 0", conv.calls) + } + if got := rec.Header().Get(ConversionHeader); got != "failed" { + t.Fatalf("%s = %q, want failed", ConversionHeader, got) + } +} + +func TestServeEbook_ContextCancelPropagates(t *testing.T) { + src := writeTemp(t, "book.mobi", "raw-mobi-source") + conv := &fakeConverter{err: context.Canceled} + h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)} + file := &models.MediaFile{ID: 6, FilePath: src, Container: "mobi"} + + rec := httptest.NewRecorder() + req := httptest.NewRequest(http.MethodGet, "/read", nil) + err := h.serveEbook(rec, req, file) + if err == nil { + t.Fatal("expected context cancellation to propagate, not fall back") + } + if rec.Header().Get(ConversionHeader) == "failed" { + t.Fatal("cancellation must not be reported as a conversion failure") + } +} diff --git a/internal/api/handlers/ebook_reader.go b/internal/api/handlers/ebook_reader.go index 6360239d..9082a80c 100644 --- a/internal/api/handlers/ebook_reader.go +++ b/internal/api/handlers/ebook_reader.go @@ -96,6 +96,9 @@ type EbookReaderHandler struct { ProgressStore EbookReaderProgressStore ConfigStore EbookReaderConfigStore AnnotationStore EbookReaderAnnotationStore + // Conversion, when set and enabled, transparently converts Kindle-family + // files to EPUB on read. Nil means the feature is off. + Conversion *EbookConversion } func NewEbookReaderHandler(authorizer *MediaFileAuthorizer) *EbookReaderHandler { @@ -130,7 +133,7 @@ func (h *EbookReaderHandler) HandleReadFile(w http.ResponseWriter, r *http.Reque return } - if err := serveEbookInline(w, r, file); err != nil { + if err := h.serveEbook(w, r, file); err != nil { if errors.Is(err, catalog.ErrItemNotFound) { writeError(w, http.StatusNotFound, "not_found", "Ebook file not found") return diff --git a/internal/api/router.go b/internal/api/router.go index cb05336c..f770bda3 100644 --- a/internal/api/router.go +++ b/internal/api/router.go @@ -515,6 +515,9 @@ func NewRouter(deps Dependencies) chi.Router { if ebookAnnotationStore != nil { ebookReaderHandler.AnnotationStore = ebookAnnotationStore } + if conv := buildEbookConversion(deps, settingsRepo); conv != nil { + ebookReaderHandler.Conversion = conv + } } catalogResourceHandler = handlers.NewCatalogResourceHandler(itemsHandler) catalogHandler = handlers.NewCatalogHandler( @@ -1993,6 +1996,7 @@ func NewRouter(deps Dependencies) chi.Router { if ebookReaderHandler != nil { r.Route("/ebooks", func(r chi.Router) { r.Use(apimw.RequireProfile) + r.Get("/capability", ebookReaderHandler.HandleConversionCapability) r.Get("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile) r.Head("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile) r.Get("/{content_id}/progress", ebookReaderHandler.HandleGetProgress) diff --git a/internal/ebookconvert/cache.go b/internal/ebookconvert/cache.go new file mode 100644 index 00000000..15d79f28 --- /dev/null +++ b/internal/ebookconvert/cache.go @@ -0,0 +1,284 @@ +package ebookconvert + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "errors" + "fmt" + "os" + "path/filepath" + "sort" + "strconv" + "strings" + "sync" + "time" + + "golang.org/x/sync/singleflight" +) + +// moduleVersion is a short fingerprint of the embedded wasm, mixed into every +// cache key so bumping mobitool.wasm transparently invalidates old conversions. +var moduleVersion = func() string { + sum := sha256.Sum256(mobitoolWasm) + return hex.EncodeToString(sum[:])[:16] +}() + +// ModuleVersion returns the embedded converter's fingerprint. Callers can mix +// it into HTTP ETags so a converter bump invalidates client/proxy caches. +func ModuleVersion() string { return moduleVersion } + +// SourceKey identifies a source file for caching. Built by the caller from the +// catalog/file metadata. Size+ModTimeNano are cheap and computed on every read; +// Checksum (the scanner's content hash, if available) hardens against a replace +// that preserves size+mtime — set it when you have it. +type SourceKey struct { + FileID int + Size int64 + ModTimeNano int64 + Checksum string // optional; scanner checksum +} + +func (k SourceKey) hash() string { + h := sha256.New() + fmt.Fprintf(h, "v1|mod=%s|id=%d|sz=%d|mt=%d|ck=%s", + moduleVersion, k.FileID, k.Size, k.ModTimeNano, k.Checksum) + return hex.EncodeToString(h.Sum(nil)) +} + +// CacheKey returns the exact identity used to cache this source's conversion +// (source identity + converter module version). Use it to derive an HTTP ETag +// so the validator and the cache entry can never disagree. +func (k SourceKey) CacheKey() string { return k.hash() } + +// CacheOptions configures a Cache. +type CacheOptions struct { + // Dir is the on-disk cache root for converted EPUBs (required). + Dir string + // MaxBytes bounds total cached EPUB size; oldest are evicted past it. + // Zero uses DefaultCacheMaxBytes. + MaxBytes int64 + // NegativeTTL is how long a DRM/failed result is remembered to avoid + // reconverting a known-bad source. Zero uses DefaultNegativeTTL. + NegativeTTL time.Duration +} + +const ( + DefaultCacheMaxBytes = 2 << 30 // 2 GiB + DefaultNegativeTTL = 6 * time.Hour +) + +// Cache wraps a Converter with an on-disk, size-bounded, singleflighted cache +// and an in-memory negative cache for DRM/failed sources. +type Cache struct { + conv *Converter + opts CacheOptions + + group singleflight.Group + + mu sync.Mutex + neg map[string]negEntry // key hash -> negative result +} + +type negEntry struct { + err error + expiry time.Time +} + +// NewCache creates the cache dir and returns a ready Cache. +func NewCache(conv *Converter, opts CacheOptions) (*Cache, error) { + if conv == nil { + return nil, ErrUnavailable + } + if opts.Dir == "" { + return nil, errors.New("ebookconvert: cache Dir is required") + } + if opts.MaxBytes <= 0 { + opts.MaxBytes = DefaultCacheMaxBytes + } + if opts.NegativeTTL <= 0 { + opts.NegativeTTL = DefaultNegativeTTL + } + if err := os.MkdirAll(opts.Dir, 0o755); err != nil { + return nil, fmt.Errorf("ebookconvert: create cache dir: %w", err) + } + return &Cache{conv: conv, opts: opts, neg: make(map[string]negEntry)}, nil +} + +// GetOrConvert returns the path to a cached EPUB for srcPath/key, converting on +// miss. Concurrent calls for the same key collapse to a single conversion. +// Returns ErrDRMProtected / ErrConversionFailed (also served from the negative +// cache), ErrConversionTimedOut, ErrSourceTooLarge, or the caller's context +// error if it cancels while waiting. +func (c *Cache) GetOrConvert(ctx context.Context, srcPath string, key SourceKey) (string, error) { + kh := key.hash() + dst := filepath.Join(c.opts.Dir, kh+".epub") + + // Fast path: already converted. Refresh mtime so the mtime-ordered budget + // eviction behaves as a real LRU — recently-read entries sort newest and + // survive eviction. + if fi, err := os.Stat(dst); err == nil && fi.Size() > 0 { + now := time.Now() + _ = os.Chtimes(dst, now, now) + return dst, nil + } + + // Negative cache: known DRM/deterministically-bad source. + if err := c.negativeLookup(kh); err != nil { + return "", err + } + + // Singleflight via DoChan with a *detached* context: coalesced callers share + // one conversion, so a single caller canceling its request must not abort + // the shared work for the others (golang.org/x/sync/singleflight.Do has no + // per-caller context, and binding it to the first caller would propagate that + // caller's cancellation to all). The conversion stays bounded by the + // Converter's own timeout + concurrency semaphore; the caller's context only + // governs how long *this* caller waits (the select below). A caller that + // gives up still leaves the conversion running to populate the cache. + ch := c.group.DoChan(kh, func() (interface{}, error) { + // Re-check after acquiring the singleflight slot (another caller may + // have just produced it). + if fi, statErr := os.Stat(dst); statErr == nil && fi.Size() > 0 { + return dst, nil + } + // Convert to a temp file in the cache dir, then atomically rename in. + tmp, tmpErr := os.CreateTemp(c.opts.Dir, "converting-*.epub") + if tmpErr != nil { + return "", fmt.Errorf("%w: cache temp: %v", ErrConversionFailed, tmpErr) + } + tmpPath := tmp.Name() + _ = tmp.Close() + _ = os.Remove(tmpPath) // Convert recreates it + + if convErr := c.conv.Convert(context.WithoutCancel(ctx), srcPath, tmpPath); convErr != nil { + _ = os.Remove(tmpPath) + c.remember(kh, convErr) + return "", convErr + } + if renErr := os.Rename(tmpPath, dst); renErr != nil { + _ = os.Remove(tmpPath) + return "", fmt.Errorf("%w: cache rename: %v", ErrConversionFailed, renErr) + } + c.enforceBudget(dst) + return dst, nil + }) + + select { + case res := <-ch: + if res.Err != nil { + return "", res.Err + } + return res.Val.(string), nil + case <-ctx.Done(): + return "", ctx.Err() + } +} + +// Lookup returns a cached conversion result WITHOUT converting on miss. It lets +// cheap callers (e.g. HEAD requests) avoid kicking off a full conversion: +// - ok == true, err == nil: a ready EPUB exists at path. +// - err != nil: the source is negatively cached (DRM/failed); path is empty. +// - ok == false, err == nil: cache miss — caller must decide whether to convert. +func (c *Cache) Lookup(key SourceKey) (path string, err error, ok bool) { + kh := key.hash() + dst := filepath.Join(c.opts.Dir, kh+".epub") + if fi, statErr := os.Stat(dst); statErr == nil && fi.Size() > 0 { + now := time.Now() + _ = os.Chtimes(dst, now, now) // a HEAD hit counts as access for LRU + return dst, nil, true + } + if negErr := c.negativeLookup(kh); negErr != nil { + return "", negErr, false + } + return "", nil, false +} + +func (c *Cache) negativeLookup(kh string) error { + c.mu.Lock() + defer c.mu.Unlock() + e, ok := c.neg[kh] + if !ok { + return nil + } + if time.Now().After(e.expiry) { + delete(c.neg, kh) + return nil + } + return e.err +} + +// remember negatively caches only *deterministic* bad outcomes — DRM +// (ErrDRMProtected) and conversion failures (ErrConversionFailed: corrupt +// source, unconvertible layout, oversize output) — so repeat reads of a +// known-bad file skip reconversion for NegativeTTL. Transient outcomes +// (ErrConversionTimedOut, context cancellation/deadline, ErrSourceTooLarge) are +// deliberately NOT cached so the next read can retry — a one-off timeout under +// load must not wedge a convertible book onto the raw-fallback path. +func (c *Cache) remember(kh string, err error) { + if !errors.Is(err, ErrDRMProtected) && !errors.Is(err, ErrConversionFailed) { + return + } + c.mu.Lock() + c.neg[kh] = negEntry{err: err, expiry: time.Now().Add(c.opts.NegativeTTL)} + c.mu.Unlock() +} + +// enforceBudget evicts the oldest (by mtime) finished cache entries until the +// total is within MaxBytes. It never evicts keep (the entry the caller is about +// to return) and ignores in-flight "converting-*" temp files, since deleting +// either would hand back a vanished path or corrupt a concurrent conversion. +// Best-effort; logs nothing (caller has no logger here). +func (c *Cache) enforceBudget(keep string) { + entries, err := os.ReadDir(c.opts.Dir) + if err != nil { + return + } + type item struct { + path string + size int64 + mod time.Time + } + var items []item + var total int64 + for _, e := range entries { + name := e.Name() + if e.IsDir() || filepath.Ext(name) != ".epub" || strings.HasPrefix(name, "converting-") { + continue // skip dirs, non-epubs, and other conversions' in-flight temps + } + fi, statErr := e.Info() + if statErr != nil { + continue + } + items = append(items, item{filepath.Join(c.opts.Dir, name), fi.Size(), fi.ModTime()}) + total += fi.Size() + } + if total <= c.opts.MaxBytes { + return + } + sort.Slice(items, func(i, j int) bool { return items[i].mod.Before(items[j].mod) }) + for _, it := range items { + if total <= c.opts.MaxBytes { + break + } + if it.path == keep { + continue // never evict the entry we're about to return + } + if os.Remove(it.path) == nil { + total -= it.size + } + } +} + +// SourceKeyFromStat builds a SourceKey from a file path + fileID, reading +// size/mtime via stat. Checksum is left empty (pass one explicitly if known). +func SourceKeyFromStat(fileID int, path string) (SourceKey, error) { + fi, err := os.Stat(path) + if err != nil { + return SourceKey{}, err + } + return SourceKey{FileID: fileID, Size: fi.Size(), ModTimeNano: fi.ModTime().UnixNano()}, nil +} + +// for tests / diagnostics +func (k SourceKey) String() string { return strconv.Itoa(k.FileID) + ":" + k.hash()[:8] } diff --git a/internal/ebookconvert/cache_test.go b/internal/ebookconvert/cache_test.go new file mode 100644 index 00000000..ac5ded7e --- /dev/null +++ b/internal/ebookconvert/cache_test.go @@ -0,0 +1,297 @@ +package ebookconvert + +import ( + "context" + "errors" + "os" + "path/filepath" + "sync" + "sync/atomic" + "testing" + "time" +) + +func newTestCache(t *testing.T, opts CacheOptions) *Cache { + t.Helper() + conv := newTestConverter(t) + if opts.Dir == "" { + opts.Dir = t.TempDir() + } + c, err := NewCache(conv, opts) + if err != nil { + t.Fatalf("NewCache: %v", err) + } + return c +} + +func keyFor(t *testing.T, fileID int, path string) SourceKey { + t.Helper() + k, err := SourceKeyFromStat(fileID, path) + if err != nil { + t.Fatal(err) + } + return k +} + +func TestCache_MissThenHit(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-ncx.mobi") + key := keyFor(t, 1, src) + + p1, err := c.GetOrConvert(context.Background(), src, key) + if err != nil { + t.Fatalf("first GetOrConvert: %v", err) + } + assertValidEpub(t, p1) + fi1, _ := os.Stat(p1) + + p2, err := c.GetOrConvert(context.Background(), src, key) + if err != nil { + t.Fatalf("second GetOrConvert: %v", err) + } + if p1 != p2 { + t.Fatalf("hit returned different path: %s vs %s", p1, p2) + } + fi2, _ := os.Stat(p2) + // A hit refreshes mtime for LRU but must NOT reconvert: the cached entry is + // the same underlying file, not a freshly converted+renamed replacement. + if !os.SameFile(fi1, fi2) { + t.Fatal("cache hit reconverted the file (replaced the cached entry)") + } +} + +func TestCache_KeyChangesOnModTime(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-ncx.mobi") + k1 := keyFor(t, 1, src) + k2 := k1 + k2.ModTimeNano = k1.ModTimeNano + 1 // simulate a re-scanned/replaced file + if k1.hash() == k2.hash() { + t.Fatal("expected different cache keys for different mtime") + } + p1, err := c.GetOrConvert(context.Background(), src, k1) + if err != nil { + t.Fatalf("convert k1: %v", err) + } + p2, err := c.GetOrConvert(context.Background(), src, k2) + if err != nil { + t.Fatalf("convert k2: %v", err) + } + if p1 == p2 { + t.Fatal("different keys must map to different cache files") + } +} + +// A transient timeout is NOT negatively cached: it would otherwise wedge a +// slow-but-convertible book onto the raw-fallback path for the whole TTL. +func TestCache_TimeoutNotNegativelyCached(t *testing.T) { + conv, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond}) + if err != nil { + t.Fatalf("NewConverter: %v", err) + } + t.Cleanup(func() { _ = conv.Close(context.Background()) }) + c, err := NewCache(conv, CacheOptions{Dir: t.TempDir()}) + if err != nil { + t.Fatalf("NewCache: %v", err) + } + src := filepath.Join("testdata", "sample-ncx.mobi") + key := keyFor(t, 1, src) + + _, err = c.GetOrConvert(context.Background(), src, key) + if !errors.Is(err, ErrConversionTimedOut) { + t.Fatalf("got %v, want ErrConversionTimedOut", err) + } + c.mu.Lock() + _, cached := c.neg[key.hash()] + c.mu.Unlock() + if cached { + t.Fatal("a transient timeout must not be negatively cached") + } +} + +// Eviction must never delete the EPUB it is about to hand back, even when the +// configured budget is smaller than a single converted file. +func TestCache_EvictionKeepsJustConverted(t *testing.T) { + c := newTestCache(t, CacheOptions{Dir: t.TempDir(), MaxBytes: 1}) + src := filepath.Join("testdata", "sample-ncx.mobi") + p, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src)) + if err != nil { + t.Fatalf("convert: %v", err) + } + if fi, statErr := os.Stat(p); statErr != nil || fi.Size() == 0 { + t.Fatalf("returned path must exist and be non-empty (eviction deleted it): stat=%v", statErr) + } +} + +// Eviction must ignore other conversions' in-flight temp files; deleting one +// would corrupt a concurrent conversion mid-flight. +func TestCache_EvictionSkipsInFlightTemps(t *testing.T) { + dir := t.TempDir() + c := newTestCache(t, CacheOptions{Dir: dir, MaxBytes: 1}) + tmp := filepath.Join(dir, "converting-inflight.epub") + if err := os.WriteFile(tmp, make([]byte, 4096), 0o644); err != nil { + t.Fatal(err) + } + old := time.Now().Add(-time.Hour) // make the temp the "oldest" candidate + _ = os.Chtimes(tmp, old, old) + + src := filepath.Join("testdata", "sample-ncx.mobi") + if _, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src)); err != nil { + t.Fatalf("convert: %v", err) + } + if _, err := os.Stat(tmp); err != nil { + t.Fatalf("eviction deleted an in-flight temp file: %v", err) + } +} + +// A cache hit refreshes mtime so the mtime-ordered budget eviction behaves as a +// real LRU (most-recently-read entries survive). +func TestCache_HitTouchesModTimeForLRU(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-ncx.mobi") + key := keyFor(t, 1, src) + p, err := c.GetOrConvert(context.Background(), src, key) + if err != nil { + t.Fatalf("convert: %v", err) + } + old := time.Now().Add(-time.Hour) + _ = os.Chtimes(p, old, old) + + if _, err := c.GetOrConvert(context.Background(), src, key); err != nil { + t.Fatalf("hit: %v", err) + } + fi, _ := os.Stat(p) + if time.Since(fi.ModTime()) > time.Minute { + t.Fatalf("cache hit did not refresh mtime for LRU; mtime is %v old", time.Since(fi.ModTime())) + } +} + +// A single caller abandoning its request (cancellation) must not abort the +// shared conversion for everyone else: the detached work still completes and +// populates the cache. +func TestCache_CallerCancelDoesNotAbortSharedWork(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-ncx.mobi") + key := keyFor(t, 1, src) + + ctx, cancel := context.WithCancel(context.Background()) + cancel() // caller bails before the conversion can finish + _, _ = c.GetOrConvert(ctx, src, key) + + dst := filepath.Join(c.opts.Dir, key.hash()+".epub") + deadline := time.Now().Add(10 * time.Second) + for time.Now().Before(deadline) { + if fi, err := os.Stat(dst); err == nil && fi.Size() > 0 { + return // shared work completed despite the caller canceling — good + } + time.Sleep(20 * time.Millisecond) + } + t.Fatal("detached conversion did not populate the cache after the caller canceled") +} + +func TestCache_NegativeCachesDRM(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-drm-v1.mobi") + key := keyFor(t, 7, src) + + err1 := mustConvertErr(t, c, src, key) + if !errors.Is(err1, ErrDRMProtected) { + t.Fatalf("got %v, want ErrDRMProtected", err1) + } + // Second call should be served from the negative cache (still DRM error). + err2 := mustConvertErr(t, c, src, key) + if !errors.Is(err2, ErrDRMProtected) { + t.Fatalf("negative-cached call: got %v, want ErrDRMProtected", err2) + } + c.mu.Lock() + _, cached := c.neg[key.hash()] + c.mu.Unlock() + if !cached { + t.Fatal("DRM result was not negatively cached") + } +} + +func TestCache_Singleflight(t *testing.T) { + c := newTestCache(t, CacheOptions{}) + src := filepath.Join("testdata", "sample-ncx.mobi") + key := keyFor(t, 3, src) + + const n = 8 + var wg sync.WaitGroup + var firstPath atomic.Value + errs := make(chan error, n) + for i := 0; i < n; i++ { + wg.Add(1) + go func() { + defer wg.Done() + p, err := c.GetOrConvert(context.Background(), src, key) + if err != nil { + errs <- err + return + } + firstPath.Store(p) + }() + } + wg.Wait() + close(errs) + for err := range errs { + t.Fatalf("concurrent GetOrConvert: %v", err) + } + if p, _ := firstPath.Load().(string); p == "" { + t.Fatal("no path produced") + } +} + +func TestCache_BudgetEviction(t *testing.T) { + dir := t.TempDir() + // Budget fits one converted EPUB (~2.4 KiB) but not two, so converting a + // second entry evicts the older one — while never evicting the entry just + // handed back. + const budget = 3000 + c := newTestCache(t, CacheOptions{Dir: dir, MaxBytes: budget}) + src := filepath.Join("testdata", "sample-ncx.mobi") + + p1, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src)) + if err != nil { + t.Fatalf("convert 1: %v", err) + } + // Force the first entry to be the oldest (the eviction victim). + old := time.Now().Add(-time.Hour) + _ = os.Chtimes(p1, old, old) + + k2 := keyFor(t, 2, src) + k2.FileID = 2 + p2, err := c.GetOrConvert(context.Background(), src, k2) + if err != nil { + t.Fatalf("convert 2: %v", err) + } + + if _, err := os.Stat(p2); err != nil { + t.Fatalf("just-converted entry must survive eviction: %v", err) + } + if _, err := os.Stat(p1); !os.IsNotExist(err) { + t.Fatalf("older entry should have been evicted, stat err = %v", err) + } + + epubs, _ := filepath.Glob(filepath.Join(dir, "*.epub")) + var total int64 + for _, e := range epubs { + fi, _ := os.Stat(e) + total += fi.Size() + } + if total > budget { + t.Fatalf("budget not enforced: total=%d > %d (%d files)", total, budget, len(epubs)) + } +} + +func TestModuleVersion_Stable(t *testing.T) { + if len(moduleVersion) != 16 { + t.Fatalf("moduleVersion len = %d, want 16", len(moduleVersion)) + } +} + +func mustConvertErr(t *testing.T, c *Cache, src string, key SourceKey) error { + t.Helper() + _, err := c.GetOrConvert(context.Background(), src, key) + return err +} diff --git a/internal/ebookconvert/converter.go b/internal/ebookconvert/converter.go new file mode 100644 index 00000000..3435ef89 --- /dev/null +++ b/internal/ebookconvert/converter.go @@ -0,0 +1,468 @@ +package ebookconvert + +import ( + "archive/zip" + "bytes" + "context" + _ "embed" + "encoding/xml" + "errors" + "fmt" + "io" + "os" + "path/filepath" + "strings" + "sync/atomic" + "time" + + "github.com/tetratelabs/wazero" + "github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1" + "github.com/tetratelabs/wazero/sys" +) + +// mobitoolWasm is libmobi's mobitool compiled to wasm32-wasi. +// Built by tools/mobitool-wasm/Dockerfile; provenance in mobitool.wasm.sha256. +// +//go:embed mobitool.wasm +var mobitoolWasm []byte + +// Guest layout. Input is mounted READ-ONLY (confined fs.FS, no `..` escape); +// output is a dedicated writable dir. NOTE: a writable wazero dir mount is not +// a hard jail (`..` traversal is possible within the server user's reach), so +// the WASM sandbox is a *memory-safety* boundary — filesystem isolation relies +// on running the server as a constrained, non-root user/container. See the +// design doc's "Filesystem sandboxing" section. +const ( + guestInDir = "/in" + guestOutDir = "/out" + guestInputName = "in.mobi" // mobitool sniffs content; extension is cosmetic + guestEpubName = "in.epub" // mobitool derives this from the input basename +) + +// mobitool diagnostic lines that mean the source is encrypted/DRM'd. Matched as +// substrings against individual output lines (not the whole blob) so book +// metadata/text containing "DRM" cannot false-positive. "Document is encrypted" +// is distinct from mobitool's "Document is not encrypted, ignoring PID/serial". +var drmMarkers = []string{ + "Document is encrypted", + "DRM key not found", + "Invalid DRM pid", + "DRM expired", + "DRM support not included", +} + +// Print Replica = fixed-layout (image) Kindle book; mobitool ignores -e and +// produces no EPUB. Detected for a clear failure message. +const printReplicaMarker = "Print Replica" + +// Options configures a Converter. Zero values fall back to sane defaults. +type Options struct { + MaxSourceBytes int64 // reject larger sources before invoking the converter + MaxOutputBytes int64 // reject a produced EPUB larger than this + Timeout time.Duration // bounds a single conversion + MaxConcurrent int // caps simultaneous conversions + MaxMemoryPages uint32 // wasm linear-memory cap (64 KiB/page) + MaxLogBytes int // cap on captured stdout+stderr each + WorkRoot string // parent for per-conversion scratch dirs ("" = os.TempDir) +} + +const ( + DefaultMaxSourceBytes = 256 << 20 // 256 MiB + DefaultMaxOutputBytes = 512 << 20 // 512 MiB + DefaultTimeout = 60 * time.Second + DefaultConcurrency = 2 + DefaultMaxMemoryPages = 16384 // 1 GiB + DefaultMaxLogBytes = 64 << 10 +) + +func (o *Options) applyDefaults() { + if o.MaxSourceBytes <= 0 { + o.MaxSourceBytes = DefaultMaxSourceBytes + } + if o.MaxOutputBytes <= 0 { + o.MaxOutputBytes = DefaultMaxOutputBytes + } + if o.Timeout <= 0 { + o.Timeout = DefaultTimeout + } + if o.MaxConcurrent <= 0 { + o.MaxConcurrent = DefaultConcurrency + } + if o.MaxMemoryPages == 0 { + o.MaxMemoryPages = DefaultMaxMemoryPages + } + if o.MaxLogBytes <= 0 { + o.MaxLogBytes = DefaultMaxLogBytes + } +} + +// Converter converts Kindle-family ebooks to EPUB in-process via wazero. +// Safe for concurrent use. The WASM module is compiled once; each conversion +// instantiates a fresh, isolated module. +type Converter struct { + runtime wazero.Runtime + compiled wazero.CompiledModule + opts Options + sem chan struct{} + closed atomic.Bool +} + +// NewConverter compiles the embedded module and returns a ready Converter. +// On failure it returns ErrUnavailable; callers must not advertise the +// conversion capability when this fails. +func NewConverter(ctx context.Context, opts Options) (*Converter, error) { + opts.applyDefaults() + + cfg := wazero.NewRuntimeConfig(). + WithCloseOnContextDone(true). + WithMemoryLimitPages(opts.MaxMemoryPages) + r := wazero.NewRuntimeWithConfig(ctx, cfg) + if _, err := wasi_snapshot_preview1.Instantiate(ctx, r); err != nil { + _ = r.Close(ctx) + return nil, fmt.Errorf("%w: instantiate wasi: %v", ErrUnavailable, err) + } + compiled, err := r.CompileModule(ctx, mobitoolWasm) + if err != nil { + _ = r.Close(ctx) + return nil, fmt.Errorf("%w: compile module: %v", ErrUnavailable, err) + } + return &Converter{ + runtime: r, + compiled: compiled, + opts: opts, + sem: make(chan struct{}, opts.MaxConcurrent), + }, nil +} + +// Close releases the runtime. Convert returns ErrUnavailable afterward. +func (c *Converter) Close(ctx context.Context) error { + if c == nil || c.runtime == nil { + return nil + } + c.closed.Store(true) + return c.runtime.Close(ctx) +} + +// Convert reads the MOBI/AZW/AZW3 at srcPath and writes a validated EPUB to +// dstPath. Returns ErrDRMProtected, ErrConversionFailed (deterministic), +// ErrConversionTimedOut (transient — the per-call timeout fired), +// ErrSourceTooLarge, ErrUnavailable, or the caller's context error on +// cancellation/deadline. +func (c *Converter) Convert(ctx context.Context, srcPath, dstPath string) error { + if c.closed.Load() { + return ErrUnavailable + } + info, statErr := os.Stat(srcPath) + if statErr != nil { + return fmt.Errorf("%w: stat source: %v", ErrConversionFailed, statErr) + } + if info.Size() > c.opts.MaxSourceBytes { + return ErrSourceTooLarge + } + + select { + case c.sem <- struct{}{}: + defer func() { <-c.sem }() + case <-ctx.Done(): + return ctx.Err() + } + + // Per-conversion scratch: in/ (read-only mount) and out/ (writable mount). + scratch, err := os.MkdirTemp(c.opts.WorkRoot, "ebookconvert-") + if err != nil { + return fmt.Errorf("%w: scratch: %v", ErrConversionFailed, err) + } + defer os.RemoveAll(scratch) + + inDir := filepath.Join(scratch, "in") + outDir := filepath.Join(scratch, "out") + if err := os.MkdirAll(inDir, 0o755); err != nil { + return fmt.Errorf("%w: in dir: %v", ErrConversionFailed, err) + } + if err := os.MkdirAll(outDir, 0o755); err != nil { + return fmt.Errorf("%w: out dir: %v", ErrConversionFailed, err) + } + if err := copyFile(srcPath, filepath.Join(inDir, guestInputName)); err != nil { + return fmt.Errorf("%w: stage input: %v", ErrConversionFailed, err) + } + + runCtx, cancel := context.WithTimeout(ctx, c.opts.Timeout) + defer cancel() + + stdout := &cappedWriter{max: c.opts.MaxLogBytes} + stderr := &cappedWriter{max: c.opts.MaxLogBytes} + fsConfig := wazero.NewFSConfig(). + WithFSMount(os.DirFS(inDir), guestInDir). // read-only, confined + WithDirMount(outDir, guestOutDir) // writable + modConfig := wazero.NewModuleConfig(). + WithArgs("mobitool", "-e", "-o", guestOutDir, guestInDir+"/"+guestInputName). + WithFSConfig(fsConfig). + WithStdout(stdout). + WithStderr(stderr). + WithSysWalltime(). + WithName("") + + mod, instErr := c.runtime.InstantiateModule(runCtx, c.compiled, modConfig) + if mod != nil { + _ = mod.Close(runCtx) + } + combined := stdout.String() + "\n" + stderr.String() + + if err := classifyRunError(ctx, runCtx, instErr, c.opts.Timeout, combined); err != nil { + return err + } + // DRM and unconvertible layouts are reported on stdout with exit 0. + if matchesAnyLine(combined, drmMarkers) { + return ErrDRMProtected + } + if lineContains(combined, printReplicaMarker) { + return fmt.Errorf("%w: print-replica (fixed-layout) book is not convertible", ErrConversionFailed) + } + + producedEpub := filepath.Join(outDir, guestEpubName) + if fi, err := os.Stat(producedEpub); err != nil { + return fmt.Errorf("%w: no EPUB produced: %s", ErrConversionFailed, trim(combined)) + } else if fi.Size() > c.opts.MaxOutputBytes { + return fmt.Errorf("%w: output EPUB exceeds %d bytes", ErrConversionFailed, c.opts.MaxOutputBytes) + } + if err := validateEpub(producedEpub); err != nil { + return fmt.Errorf("%w: %v", ErrConversionFailed, err) + } + if err := moveFile(producedEpub, dstPath); err != nil { + return fmt.Errorf("%w: deliver output: %v", ErrConversionFailed, err) + } + return nil +} + +// classifyRunError maps a wazero run error to our error taxonomy. Context +// errors (timeout/cancel) MUST be checked before a generic nonzero exit code, +// because WithCloseOnContextDone surfaces them as *sys.ExitError with special +// codes that would otherwise read as an absurd "exit code". +// +// Ordering matters: the *caller's* context is checked first so a request-level +// cancel OR deadline is propagated verbatim (never reclassified as a conversion +// verdict — that would both mislead upstream cancellation and poison the +// negative cache). Only once the caller's context is healthy do we attribute a +// deadline to our own per-call timeout, which is a transient ErrConversionTimedOut. +func classifyRunError(parent, run context.Context, instErr error, timeout time.Duration, combined string) error { + if instErr == nil { + return nil + } + // Caller's context ended (cancel or deadline) → propagate it verbatim. + if parent.Err() != nil { + return parent.Err() + } + // Our per-call timeout fired (caller's context is still healthy). + if errors.Is(run.Err(), context.DeadlineExceeded) || + errors.Is(instErr, context.DeadlineExceeded) { + return fmt.Errorf("%w: after %s", ErrConversionTimedOut, timeout) + } + if errors.Is(instErr, context.Canceled) { + return context.Canceled + } + var exitErr *sys.ExitError + if errors.As(instErr, &exitErr) { + switch exitErr.ExitCode() { + case sys.ExitCodeDeadlineExceeded: + return fmt.Errorf("%w: after %s", ErrConversionTimedOut, timeout) + case sys.ExitCodeContextCanceled: + return context.Canceled + case 0: + return nil // normal exit; output checks follow + default: + return fmt.Errorf("%w: mobitool exit %d: %s", ErrConversionFailed, exitErr.ExitCode(), trim(combined)) + } + } + return fmt.Errorf("%w: run: %v", ErrConversionFailed, instErr) +} + +// validateEpub confirms a well-formed EPUB OCF container: a zip whose first +// entry is a STORED (uncompressed) "mimetype" holding exactly +// application/epub+zip, with a META-INF/container.xml that names an OPF +// rootfile present in the archive. +func validateEpub(path string) error { + zr, err := zip.OpenReader(path) + if err != nil { + return fmt.Errorf("open epub: %w", err) + } + defer zr.Close() + + if len(zr.File) == 0 { + return errors.New("empty epub") + } + first := zr.File[0] + if first.Name != "mimetype" { + return fmt.Errorf("first entry is %q, want mimetype", first.Name) + } + if first.Method != zip.Store { + return errors.New("mimetype is not stored (uncompressed)") + } + rc, err := first.Open() + if err != nil { + return fmt.Errorf("read mimetype: %w", err) + } + mt, err := io.ReadAll(io.LimitReader(rc, 64)) + rc.Close() + if err != nil { + return fmt.Errorf("read mimetype: %w", err) + } + if string(mt) != "application/epub+zip" { + return fmt.Errorf("mimetype is %q", string(mt)) + } + + names := make(map[string]bool, len(zr.File)) + for _, f := range zr.File { + names[f.Name] = true + } + if !names["META-INF/container.xml"] { + return errors.New("missing META-INF/container.xml") + } + rootfile, err := opfRootfile(zr) + if err != nil { + return err + } + if !names[rootfile] { + return fmt.Errorf("OPF rootfile %q missing from archive", rootfile) + } + return nil +} + +func opfRootfile(zr *zip.ReadCloser) (string, error) { + for _, f := range zr.File { + if f.Name != "META-INF/container.xml" { + continue + } + rc, err := f.Open() + if err != nil { + return "", fmt.Errorf("open container.xml: %w", err) + } + data, err := io.ReadAll(io.LimitReader(rc, 64<<10)) + rc.Close() + if err != nil { + return "", fmt.Errorf("read container.xml: %w", err) + } + var c struct { + Rootfiles []struct { + FullPath string `xml:"full-path,attr"` + } `xml:"rootfiles>rootfile"` + } + if err := xml.Unmarshal(data, &c); err != nil { + return "", fmt.Errorf("parse container.xml: %w", err) + } + if len(c.Rootfiles) == 0 || c.Rootfiles[0].FullPath == "" { + return "", errors.New("container.xml has no rootfile") + } + return c.Rootfiles[0].FullPath, nil + } + return "", errors.New("missing META-INF/container.xml") +} + +// matchesAnyLine reports whether any line of s contains any of the markers. +func matchesAnyLine(s string, markers []string) bool { + for _, line := range strings.Split(s, "\n") { + for _, m := range markers { + if strings.Contains(line, m) { + return true + } + } + } + return false +} + +func lineContains(s, marker string) bool { + for _, line := range strings.Split(s, "\n") { + if strings.Contains(line, marker) { + return true + } + } + return false +} + +// cappedWriter buffers up to max bytes and silently drops the rest, always +// reporting a full write so the guest is never blocked on a full pipe. +type cappedWriter struct { + buf bytes.Buffer + max int + truncated bool +} + +func (w *cappedWriter) Write(p []byte) (int, error) { + if room := w.max - w.buf.Len(); room > 0 { + if room >= len(p) { + w.buf.Write(p) + } else { + w.buf.Write(p[:room]) + w.truncated = true + } + } else if len(p) > 0 { + w.truncated = true + } + return len(p), nil +} + +func (w *cappedWriter) String() string { return w.buf.String() } + +func copyFile(src, dst string) error { + in, err := os.Open(src) + if err != nil { + return err + } + defer in.Close() + out, err := os.Create(dst) + if err != nil { + return err + } + if _, err := io.Copy(out, in); err != nil { + out.Close() + return err + } + return out.Close() +} + +// moveFile delivers src to dst atomically: rename when possible, else copy to a +// temp file in dst's directory, fsync, and rename — never leaving a partial dst. +func moveFile(src, dst string) error { + if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil { + return err + } + if err := os.Rename(src, dst); err == nil { + return nil + } + tmp, err := os.CreateTemp(filepath.Dir(dst), ".move-*") + if err != nil { + return err + } + tmpName := tmp.Name() + in, err := os.Open(src) + if err != nil { + tmp.Close() + os.Remove(tmpName) + return err + } + _, copyErr := io.Copy(tmp, in) + in.Close() + if copyErr == nil { + copyErr = tmp.Sync() + } + if cerr := tmp.Close(); copyErr == nil { + copyErr = cerr + } + if copyErr != nil { + os.Remove(tmpName) + return copyErr + } + if err := os.Rename(tmpName, dst); err != nil { + os.Remove(tmpName) + return err + } + return os.Remove(src) +} + +func trim(s string) string { + const max = 500 + s = strings.TrimSpace(s) + if len(s) > max { + return s[:max] + "…" + } + return s +} diff --git a/internal/ebookconvert/converter_test.go b/internal/ebookconvert/converter_test.go new file mode 100644 index 00000000..ce535e74 --- /dev/null +++ b/internal/ebookconvert/converter_test.go @@ -0,0 +1,247 @@ +package ebookconvert + +import ( + "archive/zip" + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + "time" +) + +func newTestConverter(t *testing.T) *Converter { + t.Helper() + c, err := NewConverter(context.Background(), Options{}) + if err != nil { + t.Fatalf("NewConverter: %v", err) + } + t.Cleanup(func() { _ = c.Close(context.Background()) }) + return c +} + +func TestConvert_DRMFreeProducesValidEpub(t *testing.T) { + for _, fixture := range []string{"sample-ncx.mobi", "sample-cp1252.mobi"} { + t.Run(fixture, func(t *testing.T) { + c := newTestConverter(t) + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(context.Background(), filepath.Join("testdata", fixture), dst); err != nil { + t.Fatalf("Convert: %v", err) + } + // Independently re-validate the delivered file is a real EPUB. + assertValidEpub(t, dst) + }) + } +} + +func TestConvert_DRMProtectedRejected(t *testing.T) { + c := newTestConverter(t) + dst := filepath.Join(t.TempDir(), "out.epub") + err := c.Convert(context.Background(), filepath.Join("testdata", "sample-drm-v1.mobi"), dst) + if !errors.Is(err, ErrDRMProtected) { + t.Fatalf("got %v, want ErrDRMProtected", err) + } + if _, statErr := os.Stat(dst); !os.IsNotExist(statErr) { + t.Fatalf("DRM source must not produce an output file, but %s exists", dst) + } +} + +func TestConvert_OversizeRejected(t *testing.T) { + c, err := NewConverter(context.Background(), Options{MaxSourceBytes: 1024}) + if err != nil { + t.Fatalf("NewConverter: %v", err) + } + t.Cleanup(func() { _ = c.Close(context.Background()) }) + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrSourceTooLarge) { + t.Fatalf("got %v, want ErrSourceTooLarge", err) + } +} + +func TestConvert_CorruptSourceFails(t *testing.T) { + c := newTestConverter(t) + bad := filepath.Join(t.TempDir(), "bad.mobi") + if err := os.WriteFile(bad, []byte("not a mobi file at all"), 0o644); err != nil { + t.Fatal(err) + } + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(context.Background(), bad, dst); !errors.Is(err, ErrConversionFailed) { + t.Fatalf("got %v, want ErrConversionFailed", err) + } +} + +func TestConvert_MissingSourceFails(t *testing.T) { + c := newTestConverter(t) + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(context.Background(), "testdata/does-not-exist.mobi", dst); !errors.Is(err, ErrConversionFailed) { + t.Fatalf("got %v, want ErrConversionFailed", err) + } +} + +func TestConvert_Concurrent(t *testing.T) { + c := newTestConverter(t) + const n = 6 + errs := make(chan error, n) + for i := 0; i < n; i++ { + go func(i int) { + dst := filepath.Join(t.TempDir(), "c.epub") + errs <- c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst) + }(i) + } + for i := 0; i < n; i++ { + if err := <-errs; err != nil { + t.Fatalf("concurrent Convert: %v", err) + } + } +} + +func TestConvert_ContextCancelled(t *testing.T) { + c := newTestConverter(t) + ctx, cancel := context.WithCancel(context.Background()) + cancel() // already canceled before the conversion starts + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(ctx, filepath.Join("testdata", "sample-ncx.mobi"), dst); err == nil { + t.Fatal("expected error on canceled context") + } +} + +func TestValidateEpub_RejectsNonEpubZip(t *testing.T) { + // A zip that is not an EPUB (no mimetype-first) must be rejected. + p := filepath.Join(t.TempDir(), "plain.zip") + f, err := os.Create(p) + if err != nil { + t.Fatal(err) + } + zw := zip.NewWriter(f) + w, _ := zw.Create("hello.txt") + _, _ = w.Write([]byte("hi")) + _ = zw.Close() + _ = f.Close() + if err := validateEpub(p); err == nil { + t.Fatal("expected validateEpub to reject a non-EPUB zip") + } +} + +func TestValidateEpub_RejectsDeflatedMimetype(t *testing.T) { + // mimetype present + correct content but DEFLATED (not stored) -> reject. + p := filepath.Join(t.TempDir(), "bad.epub") + f, err := os.Create(p) + if err != nil { + t.Fatal(err) + } + zw := zip.NewWriter(f) + w, _ := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Deflate}) + _, _ = w.Write([]byte("application/epub+zip")) + _ = zw.Close() + _ = f.Close() + if err := validateEpub(p); err == nil { + t.Fatal("expected rejection of deflated mimetype") + } +} + +func TestConvert_AfterCloseReturnsUnavailable(t *testing.T) { + c, err := NewConverter(context.Background(), Options{}) + if err != nil { + t.Fatalf("NewConverter: %v", err) + } + _ = c.Close(context.Background()) + dst := filepath.Join(t.TempDir(), "out.epub") + if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrUnavailable) { + t.Fatalf("got %v, want ErrUnavailable", err) + } +} + +func TestConvert_TimeoutClassifiedNotRawExit(t *testing.T) { + // A 1ns timeout must surface as ErrConversionTimedOut (a transient verdict, + // distinct from a deterministic failure), never as a bogus "mobitool exit + // " code. + c, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond}) + if err != nil { + t.Fatalf("NewConverter: %v", err) + } + t.Cleanup(func() { _ = c.Close(context.Background()) }) + dst := filepath.Join(t.TempDir(), "out.epub") + err = c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst) + if !errors.Is(err, ErrConversionTimedOut) { + t.Fatalf("got %v, want ErrConversionTimedOut", err) + } + if errors.Is(err, ErrConversionFailed) { + t.Fatalf("a timeout must not also satisfy ErrConversionFailed (it would be negatively cached): %v", err) + } + if strings.Contains(err.Error(), "exit ") { + t.Fatalf("timeout leaked a raw exit code: %v", err) + } +} + +// classifyRunError must propagate a caller's deadline as context.DeadlineExceeded +// (CR review): reclassifying it as a conversion failure breaks upstream +// cancellation handling and would poison the negative cache. +func TestClassifyRunError_ParentDeadlinePropagates(t *testing.T) { + parent, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour)) + defer cancel() + run := parent // run derives from parent, so it is also past-deadline + err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output") + if !errors.Is(err, context.DeadlineExceeded) { + t.Fatalf("got %v, want context.DeadlineExceeded propagated", err) + } + if errors.Is(err, ErrConversionFailed) || errors.Is(err, ErrConversionTimedOut) { + t.Fatalf("a caller deadline must not be reclassified as a conversion verdict: %v", err) + } +} + +// When the caller's context is healthy but our own per-call timeout fired, the +// result is the transient ErrConversionTimedOut sentinel (not ErrConversionFailed). +func TestClassifyRunError_OwnTimeoutIsTimedOutSentinel(t *testing.T) { + parent := context.Background() // caller healthy + run, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour)) + defer cancel() + err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output") + if !errors.Is(err, ErrConversionTimedOut) { + t.Fatalf("got %v, want ErrConversionTimedOut", err) + } + if errors.Is(err, ErrConversionFailed) { + t.Fatalf("own timeout must not satisfy ErrConversionFailed: %v", err) + } +} + +func assertValidEpub(t *testing.T, path string) { + t.Helper() + zr, err := zip.OpenReader(path) + if err != nil { + t.Fatalf("open epub: %v", err) + } + defer zr.Close() + if len(zr.File) == 0 || zr.File[0].Name != "mimetype" { + t.Fatalf("first entry not mimetype") + } + rc, _ := zr.File[0].Open() + mt, _ := io.ReadAll(rc) + rc.Close() + if strings.TrimSpace(string(mt)) != "application/epub+zip" { + t.Fatalf("bad mimetype %q", mt) + } + var hasContainer, hasContent bool + for _, fl := range zr.File { + if fl.Name == "META-INF/container.xml" { + hasContainer = true + } + if strings.HasPrefix(fl.Name, "OEBPS/") { + hasContent = true + } + } + if !hasContainer { + t.Fatal("missing META-INF/container.xml") + } + if !hasContent { + t.Fatal("no OEBPS content") + } +} + +// Guard so the default timeout constant stays sane if edited. +func TestDefaults(t *testing.T) { + if DefaultTimeout < time.Second { + t.Fatal("DefaultTimeout too small") + } +} diff --git a/internal/ebookconvert/errors.go b/internal/ebookconvert/errors.go new file mode 100644 index 00000000..437f0310 --- /dev/null +++ b/internal/ebookconvert/errors.go @@ -0,0 +1,35 @@ +// Package ebookconvert converts Kindle-family ebooks (MOBI/AZW/AZW3) to EPUB +// in-process, by running libmobi's mobitool compiled to wasm32-wasi on the +// wazero runtime. No cgo, no external binary; untrusted input is parsed inside +// the WASM sandbox. See docs/.../2026-06-17-kindle-epub-conversion-design.md. +package ebookconvert + +import "errors" + +var ( + // ErrDRMProtected means the source is encrypted/DRM'd and cannot be + // converted without a device key. Callers should surface the + // "open externally" path; the source is never usable in-app. + ErrDRMProtected = errors.New("ebookconvert: source is DRM-protected") + + // ErrConversionFailed means mobitool failed or produced output that is + // not a valid EPUB (corrupt source, unexpected format, internal error). + // It is a *deterministic* verdict for a given input — the same bytes will + // fail the same way — so the cache may remember it negatively. + ErrConversionFailed = errors.New("ebookconvert: conversion failed") + + // ErrConversionTimedOut means the converter's own per-call timeout fired. + // Unlike ErrConversionFailed it is *transient* (it reflects load/size, not + // the input being unconvertible), so it is NOT negatively cached and the + // next read may retry. It deliberately does not wrap ErrConversionFailed. + ErrConversionTimedOut = errors.New("ebookconvert: conversion timed out") + + // ErrSourceTooLarge means the source exceeds the configured size cap and + // was rejected before invoking the converter. + ErrSourceTooLarge = errors.New("ebookconvert: source exceeds max size") + + // ErrUnavailable means the converter could not be initialized (the + // embedded module failed to compile/instantiate). The capability must + // not be advertised in this state. + ErrUnavailable = errors.New("ebookconvert: converter unavailable") +) diff --git a/internal/ebookconvert/mobitool.wasm b/internal/ebookconvert/mobitool.wasm new file mode 100755 index 00000000..a56eb87c Binary files /dev/null and b/internal/ebookconvert/mobitool.wasm differ diff --git a/internal/ebookconvert/mobitool.wasm.sha256 b/internal/ebookconvert/mobitool.wasm.sha256 new file mode 100644 index 00000000..2ef3ea59 --- /dev/null +++ b/internal/ebookconvert/mobitool.wasm.sha256 @@ -0,0 +1 @@ +38a544c5f516a6b51c9eba21f7f06e964de55aabdd05c3bbe7825f08e6f88303 /out/mobitool.wasm diff --git a/internal/ebookconvert/provenance_test.go b/internal/ebookconvert/provenance_test.go new file mode 100644 index 00000000..d9b85ca0 --- /dev/null +++ b/internal/ebookconvert/provenance_test.go @@ -0,0 +1,33 @@ +package ebookconvert + +import ( + "crypto/sha256" + "encoding/hex" + "os" + "strings" + "testing" +) + +// The committed mobitool.wasm must match the sha256 recorded next to it. This +// is the cheap half of the artifact guardrail: together with the real fixture +// conversions in converter_test.go (the "smoke test"), it ensures a careless +// artifact swap or a corrupted checkout is caught in CI instead of shipping a +// mystery binary. The sidecar is sha256sum format (` `); we compare +// only the hash, since the recorded path is build-environment specific. +func TestEmbeddedWasmMatchesRecordedSHA256(t *testing.T) { + raw, err := os.ReadFile("mobitool.wasm.sha256") + if err != nil { + t.Fatalf("read sha256 sidecar: %v", err) + } + fields := strings.Fields(string(raw)) + if len(fields) == 0 { + t.Fatal("mobitool.wasm.sha256 is empty") + } + want := strings.ToLower(fields[0]) + + sum := sha256.Sum256(mobitoolWasm) + got := hex.EncodeToString(sum[:]) + if got != want { + t.Fatalf("embedded mobitool.wasm sha256 = %s, recorded = %s — artifact and provenance disagree", got, want) + } +} diff --git a/internal/ebookconvert/testdata/README.md b/internal/ebookconvert/testdata/README.md new file mode 100644 index 00000000..abb421b1 --- /dev/null +++ b/internal/ebookconvert/testdata/README.md @@ -0,0 +1,12 @@ +# Test fixtures + +These are libmobi's own test sample files, vendored for converter unit tests. + +Source: https://github.com/bfabiszewski/libmobi `tests/samples/` (LGPL-3.0; the +samples are project test data authored by the libmobi project). + +| File | Purpose | +|------|---------| +| `sample-ncx.mobi` | DRM-free KF8/MOBI6 hybrid — happy-path conversion | +| `sample-cp1252.mobi` | DRM-free cp1252-encoded — second happy-path encoding | +| `sample-drm-v1.mobi` | Encrypted source — exercises the DRM rejection path | diff --git a/internal/ebookconvert/testdata/sample-cp1252.mobi b/internal/ebookconvert/testdata/sample-cp1252.mobi new file mode 100644 index 00000000..621a5514 Binary files /dev/null and b/internal/ebookconvert/testdata/sample-cp1252.mobi differ diff --git a/internal/ebookconvert/testdata/sample-drm-v1.mobi b/internal/ebookconvert/testdata/sample-drm-v1.mobi new file mode 100644 index 00000000..760c6691 Binary files /dev/null and b/internal/ebookconvert/testdata/sample-drm-v1.mobi differ diff --git a/internal/ebookconvert/testdata/sample-ncx.mobi b/internal/ebookconvert/testdata/sample-ncx.mobi new file mode 100644 index 00000000..7a0d92ea Binary files /dev/null and b/internal/ebookconvert/testdata/sample-ncx.mobi differ diff --git a/tools/mobitool-wasm/Dockerfile b/tools/mobitool-wasm/Dockerfile new file mode 100644 index 00000000..aec82446 --- /dev/null +++ b/tools/mobitool-wasm/Dockerfile @@ -0,0 +1,84 @@ +# Builds `mobitool.wasm` — libmobi's mobitool compiled to wasm32-wasi — used by +# the in-process Kindle->EPUB converter (internal/ebookconvert) via wazero. +# +# The output is architecture-independent wasm32 bytecode: build it on any host +# (CI is amd64), run it on any server arch through wazero (pure Go). +# +# Build + extract the artifact: +# docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm +# id=$(docker create mobitool-wasm); docker cp "$id:/out/mobitool.wasm" \ +# internal/ebookconvert/mobitool.wasm; docker rm "$id" +# +# Pins (bump deliberately; the converter cache key includes these): +# wasi-sdk 25.0 · libmobi 906274205c11944b628da1c553b255acb1af7c55 · zlib 1.3.1 +# +# Proven build config (see ../../docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md): +# --with-libxml2=no -> internal xmlwriter provides OPF/EPUB support (-e) +# zlib built to wasm -> avoids libmobi's bundled-miniz duplicate-symbol clash +# with mobitool's own miniz (EPUB zip) at wasm-ld time. + +FROM debian:bookworm-slim AS build + +ARG WASI_SDK_VERSION=25 +ARG LIBMOBI_COMMIT=906274205c11944b628da1c553b255acb1af7c55 +ARG ZLIB_VERSION=1.3.1 + +RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \ + build-essential autoconf automake libtool pkg-config git curl ca-certificates xz-utils \ + && rm -rf /var/lib/apt/lists/* + +# --- wasi-sdk (clang + wasm32-wasi sysroot) --- +RUN cd /opt \ + && curl -fsSL -o wasi-sdk.tar.gz \ + "https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux.tar.gz" \ + && tar xzf wasi-sdk.tar.gz && mv "wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux" wasi-sdk && rm wasi-sdk.tar.gz +ENV WASI=/opt/wasi-sdk +ENV SYSROOT=/opt/wasi-sdk/share/wasi-sysroot + +# --- zlib -> wasm (compile sources directly; zlib's configure runs target binaries) --- +RUN cd /opt \ + && curl -fsSL -o zlib.tar.gz "https://github.com/madler/zlib/releases/download/v${ZLIB_VERSION}/zlib-${ZLIB_VERSION}.tar.gz" \ + && tar xzf zlib.tar.gz && cd "zlib-${ZLIB_VERSION}" \ + && for s in adler32 compress crc32 deflate gzclose gzlib gzread gzwrite infback inffast inflate inftrees trees uncompr zutil; do \ + "$WASI/bin/clang" -O2 --sysroot="$SYSROOT" -DHAVE_UNISTD_H -c "$s.c" -o "$s.o"; done \ + && "$WASI/bin/llvm-ar" rcs libz.a *.o \ + && mkdir -p /opt/zlibwasm/lib /opt/zlibwasm/include \ + && cp libz.a /opt/zlibwasm/lib/ && cp zlib.h zconf.h /opt/zlibwasm/include/ + +# --- libmobi + mobitool -> wasm --- +RUN mkdir -p /work && cd /work \ + && git clone https://github.com/bfabiszewski/libmobi.git && cd libmobi \ + && git checkout "${LIBMOBI_COMMIT}" \ + && ./autogen.sh \ + && ./configure --host=wasm32-wasi \ + --with-libxml2=no --disable-shared --enable-static \ + CC="$WASI/bin/clang" AR="$WASI/bin/llvm-ar" RANLIB="$WASI/bin/llvm-ranlib" \ + CFLAGS="-O2 --sysroot=$SYSROOT -I/opt/zlibwasm/include" \ + CPPFLAGS="-I/opt/zlibwasm/include" \ + LDFLAGS="--sysroot=$SYSROOT -L/opt/zlibwasm/lib" \ + LIBS="-lz" \ + && make -j"$(nproc)" \ + && mkdir -p /out && cp tools/mobitool /out/mobitool.wasm \ + && sha256sum /out/mobitool.wasm | tee /out/mobitool.wasm.sha256 + +# Smoke test: convert a bundled libmobi sample to EPUB using a pinned, +# checksum-verified wasmtime — NOT the `curl | bash` convenience installer, which +# is documented as interactive-only and unsafe/non-deterministic for builds — so +# the build fails loudly if the wasm can't actually convert. +ARG WASMTIME_VERSION=27.0.0 +ARG WASMTIME_SHA256=74678f6ec49d8b858ca5181d5492b967245f6c072b5763ea14a75eee1ad3c2d7 +RUN cd /opt \ + && curl -fsSL -o wasmtime.tar.xz \ + "https://github.com/bytecodealliance/wasmtime/releases/download/v${WASMTIME_VERSION}/wasmtime-v${WASMTIME_VERSION}-x86_64-linux.tar.xz" \ + && echo "${WASMTIME_SHA256} wasmtime.tar.xz" | sha256sum -c - \ + && tar xJf wasmtime.tar.xz && rm wasmtime.tar.xz \ + && WASMTIME="/opt/wasmtime-v${WASMTIME_VERSION}-x86_64-linux/wasmtime" \ + && mkdir -p /smoke/out \ + && cp /work/libmobi/tests/samples/sample-ncx.mobi /smoke/in.mobi \ + && "$WASMTIME" run --dir=/smoke::/smoke /out/mobitool.wasm -e -o /smoke/out /smoke/in.mobi \ + && test -f /smoke/out/in.epub \ + && echo "SMOKE OK: $(wc -c < /smoke/out/in.epub) byte EPUB produced" + +FROM scratch AS artifact +COPY --from=build /out/mobitool.wasm /mobitool.wasm +COPY --from=build /out/mobitool.wasm.sha256 /mobitool.wasm.sha256 diff --git a/tools/mobitool-wasm/README.md b/tools/mobitool-wasm/README.md new file mode 100644 index 00000000..100dcfa9 --- /dev/null +++ b/tools/mobitool-wasm/README.md @@ -0,0 +1,64 @@ +# mobitool.wasm — Kindle→EPUB converter module + +`mobitool.wasm` is [libmobi](https://github.com/bfabiszewski/libmobi)'s `mobitool` +compiled to `wasm32-wasi`. The server runs it **in-process** via +[`wazero`](https://github.com/tetratelabs/wazero) (pure Go) to convert +MOBI/AZW/AZW3 ebooks to EPUB on demand. See the design doc: +`docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md`. + +## Why this shape +- **No cgo, no external binary.** The `.wasm` is `go:embed`-ed; the Go build stays + `CGO_ENABLED=0` and cross-compiles trivially. +- **Architecture-independent.** wasm32 bytecode runs identically on amd64/arm64 + servers through wazero. Build once (CI, amd64), run anywhere. +- **Sandboxed.** Untrusted ebook bytes are parsed inside the WASM sandbox with + only a per-conversion scratch dir mounted — a libmobi memory bug on a malicious + file cannot reach the host. + +## Pinned versions + +| Component | Version / commit | +|-------------------|--------------------------------------------| +| wasi-sdk | 25.0 | +| libmobi | `906274205c11944b628da1c553b255acb1af7c55` | +| zlib | 1.3.1 (compiled to wasm) | +| wasmtime (smoke) | 27.0.0 (build-time smoke test only) | + +License: libmobi is **LGPL-3.0-or-later**; shipping the unmodified compiled +artifact + this build recipe satisfies the relink obligation. zlib is zlib-license. + +## Build config notes (validated by spike, 2026-06-17 on linux/amd64) +- `--with-libxml2=no` → libmobi's **internal xmlwriter** provides OPF/EPUB output, + so the `-e` (create EPUB) path works with no libxml2 dependency. +- libmobi is built **against a wasm-compiled zlib** (not `--with-zlib=no`). Forcing + `--with-zlib=no` makes libmobi vendor its own miniz, which then collides with + `mobitool`'s separate miniz (used for EPUB zip creation) — `wasm-ld` rejects the + duplicate symbols. Real zlib for the library + the tool's miniz for zip = no clash. + +## Rebuild + install the artifact +```sh +docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm +id=$(docker create mobitool-wasm) +docker cp "$id:/mobitool.wasm" internal/ebookconvert/mobitool.wasm +docker cp "$id:/mobitool.wasm.sha256" internal/ebookconvert/mobitool.wasm.sha256 +docker rm "$id" +``` +The build runs a **smoke conversion** of a bundled libmobi sample and fails if no +EPUB is produced. `mobitool -v` embeds `__DATE__`/`__TIME__`, so rebuilds are not +byte-for-byte reproducible — rely on the smoke test + recorded sha256, not a +bit-identical rebuild check. + +## Runtime behavior characterized by the spike +- Converts MOBI6, KF8/AZW3 hybrids, HUFF/CDIC-compressed, and unicode samples to + well-formed EPUB (`mimetype` first = `application/epub+zip`, `META-INF/container.xml`, + `OEBPS/*`). +- **DRM:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0** + and writes an EPUB (garbage content when it lacks the key). So the converter must + capture stdout and validate the output is a usable EPUB — exit code alone is not a + reliable DRM/failure signal. + - ⚠️ **Brittleness note:** DRM/print-replica detection matches the **English + stdout strings** mobitool emits (`drmMarkers` / `printReplicaMarker` in + `internal/ebookconvert/converter.go`). When bumping the pinned libmobi commit, + re-check those messages still match — a wording change silently downgrades a + DRM book to a generic "no EPUB produced" failure (it still falls back to the raw + original safely, just without the explicit DRM signal).