Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171)
* Kindle->EPUB conversion: design + proven wasm build pipeline
Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.
This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
9 review fixes folded in: failure contract, strong cache key + negative
cache, wazero command-module specifics, FS-sandbox tightening,
double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
built on amd64. go:embed target for the converter package (next).
Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.
Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: converter core + cache (Codex-reviewed)
internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.
18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).
Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
FS isolation ultimately relies on running as a non-root user (memory-safety
is the WASM boundary). validateEpub now requires STORED mimetype + verifies
the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint
Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.
- handlers.EbookConversion (converter + per-request flag predicate) on the read
handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
served_format, header contract}; enabled only when flag on AND converter
wired (double gate) so the Android client can decide whether to flip
mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
if it fails), cache dir is a sibling of TranscodeDir, flag read per request.
Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification
Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.
Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
the per-call timeout as transient and propagate a caller's cancel/deadline
verbatim instead of reclassifying it as a conversion failure. remember() now
only caches deterministic verdicts (DRM / failed), so a one-off timeout under
load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
context.WithoutCancel), so one caller cancelling no longer aborts the shared
work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
not FIFO.
Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
a negatively-cached source serves the failed contract, a miss advertises the
converted representation cheaply without triggering a (minute-long, ~1 GiB)
conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
capability endpoint no longer hit the DB per request.
Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
+ provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).
Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: remove ebookconvert workflow
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
This commit is contained in:
RXWatcher
committed by
GitHub
GitHub
co-authored by
Claude Code
Quick
Claude Code
Quick
parent
9b111649f3
commit
e99079abf8
@@ -0,0 +1,96 @@
|
||||
# Server-side Kindle→EPUB conversion (MOBI / AZW / AZW3) — design
|
||||
|
||||
**Goal:** Let the silo-server render Kindle-family ebooks (mobi, azw, azw3) in the Android in-app reader by converting them to EPUB **server-side, in-process, on demand**, served transparently through the existing read endpoint. No external converter binary, no cgo.
|
||||
|
||||
**Status:** Approach approved by the user: convert via **libmobi**, run **built-in** to the server as **WASM executed by `wazero`** (a pure-Go runtime). On-demand + cached, admin-gated, DRM-free only. This is the MOBI half of the "all 9 server formats in-app" goal; the CBR (client-side) half is a separate Android spec.
|
||||
|
||||
> **Linchpin PROVEN (build spike, 2026-06-17, linux/amd64).** libmobi + `mobitool` cross-compile to `wasm32-wasi` (wasi-sdk 25, libmobi `9062742`, zlib 1.3.1 built to wasm); `-e` (create EPUB) is present; the module converts real MOBI6 / KF8-hybrid / HUFF-CDIC / unicode samples to **well-formed EPUB** (`mimetype`=`application/epub+zip` first, `META-INF/container.xml`, `OEBPS/*`). Verified end-to-end under **wazero** (the prod runtime) with WASI preopen + argv + `_start` + exit handling. Reproducible build stage committed at `tools/mobitool-wasm/Dockerfile` (with a smoke-conversion gate). Build-config gotcha resolved: build libmobi against a **wasm-compiled zlib** — `--with-zlib=no` makes libmobi vendor miniz that collides with mobitool's own miniz at `wasm-ld`. **DRM finding:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0** and writes a (garbage) EPUB — so detect failure by capturing stdout + validating output, never by exit code alone.
|
||||
|
||||
## Why this shape
|
||||
|
||||
- Pure-Go MOBI/KF8 *parsers* don't exist at usable quality (the Go MOBI packages only *write*). Calibre is the gold standard but ~300MB+ of Python/Qt — rejected as too heavy. **libmobi** (mature C library behind the `mobitool` CLI) handles MOBI6 + KF8/AZW3 + AZW, decompression, and EPUB generation.
|
||||
- Rather than shell out to an external `mobitool` binary (a deploy moving part) or bind via cgo (forces `CGO_ENABLED=1`, a C toolchain, painful cross-compile, and an LGPLv3 static-link obligation), we compile **`mobitool` to a WASI `.wasm`** once and **run it in-process with `wazero`**.
|
||||
- This keeps the main Go build **`CGO_ENABLED=0`** and pure; one **architecture-independent** `.wasm` runs on amd64/arm64 servers alike; the `.wasm` is `go:embed`-ed into the binary (self-contained); libmobi's LGPL is cleanly isolated as a separate embedded artifact; and — a real bonus — **untrusted ebook bytes are parsed inside the WASM sandbox**, so a memory-safety bug in libmobi on a malicious file cannot reach the host (only the mapped temp dir).
|
||||
|
||||
## Components
|
||||
|
||||
### 1. The WASM module (build-time artifact)
|
||||
- Compile libmobi + its `mobitool` tool to `wasm32-wasi` using the **wasi-sdk** (clang). Output: `mobitool.wasm`.
|
||||
- Build libmobi **without libxml2** (`--with-libxml2=no` / `--disable-xmlwriter` as available) so the module has no external native deps; mobitool's EPUB writer uses a bundled miniz, which is portable C and compiles to WASI cleanly. (Feasibility of the no-libxml2 build for the `-e` path is the #1 thing to verify in the build spike — see Risks.)
|
||||
- Build it in a **dedicated Docker build stage** (or a committed `Makefile`/script under `tools/mobitool-wasm/`). **Commit the resulting `mobitool.wasm`** into the repo so the normal Go build does not need the wasi-sdk — it only `go:embed`s the prebuilt artifact. Rebuild the `.wasm` only when bumping libmobi; record the libmobi version + build flags next to it.
|
||||
|
||||
### 2. `internal/ebookconvert` (new package) — the in-process converter
|
||||
```go
|
||||
// Converter converts a Kindle-family ebook file to EPUB bytes, in-process via wazero.
|
||||
type Converter interface {
|
||||
// Convert reads srcPath (mobi/azw/azw3) and returns the path to a converted .epub.
|
||||
// Returns ErrDRMProtected / ErrUnsupported / ErrConversionFailed as appropriate.
|
||||
Convert(ctx context.Context, srcPath string) (epubPath string, err error)
|
||||
}
|
||||
```
|
||||
- `//go:embed mobitool.wasm` → compile the module **once** at startup with `wazero.NewRuntime` + `CompileModule` (the compiled module is cached and reused; only cheap instantiation happens per conversion).
|
||||
- **wazero command-module specifics** (from Codex review): `wasi_snapshot_preview1.MustInstantiate` the host module once; `Runtime.CompileModule` once at startup; per conversion instantiate with `ModuleConfig.WithArgs(...)` (argv ≈ `mobitool -e -o <outdir> <input>`), `WithFSConfig`, `WithStdout`+`WithStderr` (capture **both** — `mobitool.c` prints many failures with `printf` to stdout, not stderr), and the default `_start`. Use `WithName("")` (or a unique name) for repeated instantiation. Non-zero exits surface as `*sys.ExitError`. Honor ctx timeout via `RuntimeConfig.WithCloseOnContextDone(true)`.
|
||||
- **Filesystem sandboxing** (from Codex review): wazero's `WithDirMount`/`WithReadOnlyDirMount` still permit relative `../../` traversal *within host permissions*, so **never preopen the library directory or the cache root**. Per conversion: copy the source into a private scratch tree, mount input **read-only via an `fs.FS`** (`WithFSMount(os.DirFS(in), …)` — confined, no `..` escape) and a dedicated empty **writable** output dir (`WithDirMount`). **The WASM sandbox is a memory-safety boundary, not a hard filesystem jail** — a writable dir mount can still be traversed by a compromised guest within the server user's reach. Filesystem isolation therefore depends on a **deployment requirement: run silo-server as a constrained, non-root service user (the container already does), so the converter's blast radius is minimal.** This is documented here and in `internal/ebookconvert/converter.go`.
|
||||
- Enforce a wall-clock **timeout** (ctx), a **max source size**, and a **concurrency semaphore** (N simultaneous conversions). WASM memory is bounded by wazero config. **Resource note:** the defaults are `DefaultMaxMemoryPages` = 1 GiB linear memory per conversion × `DefaultConcurrency` = 2, i.e. up to ~2 GiB of wasm memory under saturation (plus the 256 MiB source / 512 MiB output caps). Generous for a small server; tune `ebookconvert.Options` for tight deployments.
|
||||
- Map mobitool's exit code / captured stdout+stderr to typed errors. **Deterministic** verdicts (cacheable): DRM-protected input → `ErrDRMProtected`; corrupt / unconvertible layout / oversize-output → `ErrConversionFailed`. **Transient** verdicts (never negatively cached, retried next read): the per-call timeout firing → `ErrConversionTimedOut`; source over the size cap → `ErrSourceTooLarge`; caller cancellation/deadline → the context error propagated verbatim.
|
||||
|
||||
### 3. Conversion cache (on-demand, keyed)
|
||||
- Cache converted EPUBs on disk under a configured cache root, keyed by **strong source identity + converter identity** — `file_id + size + mtime_ns + content hash` (use the scanner checksum if one exists) **plus** the converter **module fingerprint** (`moduleVersion`: a sha256 of the embedded `mobitool.wasm`, which already pins the libmobi commit, build flags, and wasi-sdk used to produce it). (Plain `path|size|mtime` can false-hit when a file is replaced with preserved mtime/size — Codex.) A stale key misses and reconverts.
|
||||
- **Singleflight** per key so concurrent reads of the same just-opened book convert once. The shared conversion runs on a **detached context** (decoupled from any one caller), so a single caller cancelling its request never aborts the work for the others.
|
||||
- **Negative caching**: remember only **deterministic-bad** sources — DRM (`ErrDRMProtected`) and conversion failures (`ErrConversionFailed`) — so every reader open does not reconvert a known-bad file. Transient outcomes (timeout, cancellation, oversize source) are **not** cached so they retry.
|
||||
- A cache **hit refreshes the entry's mtime**, so the mtime-ordered budget eviction acts as a real LRU; eviction never deletes the entry it is handing back or another conversion's in-flight temp.
|
||||
- Bounded by an LRU / total-size budget with opportunistic eviction (mirrors the existing reader-file/image cache conventions in the codebase).
|
||||
|
||||
### 4. Wiring into the read path
|
||||
- `serveEbookInline` (`internal/api/handlers/ebook_reader.go:602`) currently `os.Open`s `file.FilePath` and `http.ServeContent`s it with `ebookMimeType(...)`.
|
||||
- Change: when the source format is **kindle-family AND conversion is enabled**, resolve the converted EPUB via the cache (convert-on-miss), then serve **that** file with `Content-Type: application/epub+zip`. All other formats are served exactly as today (raw, byte-range, unchanged).
|
||||
- Byte-range/`http.ServeContent` semantics are preserved — we serve the cached EPUB `*os.File` the same way, so resume/range still works.
|
||||
- **Response headers** (from Codex review): `Content-Type: application/epub+zip`, `Content-Disposition` with an `.epub` filename, a strong `ETag` derived from the conversion cache key, and — because the *same URL* can return raw or converted bytes depending on the setting/failure — `Cache-Control: private, max-age=0, must-revalidate` (or `no-store`) to prevent stale client/proxy caching of the wrong representation.
|
||||
- **HEAD is cache-only.** A `HEAD /read` must never trigger a (possibly minute-long, ~1 GiB) conversion. It answers from the cache: a hit serves the real converted headers + `Content-Length`; a negatively-cached source serves the `failed` contract; a miss advertises the converted headers (`Content-Type`, the cache-key `ETag`, `X-Silo-Ebook-Conversion: converted`) **optimistically with no body and no conversion** — the subsequent `GET` produces the body and the authoritative DRM/failure verdict, which then populates the cache that later HEADs read. *(Client note: a HEAD miss intentionally omits `Content-Length`; coordinate with the Android/Apple clients if any relies on it.)*
|
||||
|
||||
### 5. Admin gate + client capability
|
||||
- An **admin-toggleable setting** (`ebook.kindle_conversion_enabled`, default off) gates the whole feature, via the existing server settings mechanism (`internal/api/handlers/settings.go` / `admin.go` `HandleUpdateSetting`). The flag is read with a short TTL cache so the hot read path (and the capability endpoint clients poll) does not hit the DB per request, while toggles still apply within seconds without a restart. When off: kindle-family files serve raw as today and the capability is not advertised.
|
||||
- **Two gates to advertise the capability** (from Codex review): advertise conversion availability to clients **only when** (a) the admin flag is on **and** (b) the embedded WASM module compiled + instantiated successfully at startup. The Android client flips mobi/azw/azw3 from `ExternalOnly`→`InApp` **only when the flag is advertised**; otherwise it keeps today's external-open fallback. (Android-side change is small and tracked separately; this spec is server-side.)
|
||||
- **Capability endpoint (implemented):** `GET /api/v1/ebooks/capability` → `{enabled, source_formats:["mobi","azw","azw3"], served_format:"epub", header:"X-Silo-Ebook-Conversion", header_failed_value:"failed"}`. `enabled` is true only when both gates hold (`EbookConversion.active`). The read response carries `X-Silo-Ebook-Conversion: converted|failed` so the client knows whether the body is EPUB or the raw original (→ external-open). ETag on the converted response is derived from the **exact conversion cache key** (`SourceKey.CacheKey()`) so a validator can never disagree with the cache; the failed/raw fallback is `Cache-Control: no-store` since the same URL flips representations with the flag.
|
||||
|
||||
## Data flow (read of a `.azw3`, conversion enabled)
|
||||
1. `GET /api/v1/ebooks/{cid}/files/{fid}/read` → authorize → `file` is `azw3`, conversion enabled.
|
||||
2. Cache lookup by source identity+version. Hit → serve cached `.epub`. Miss → singleflight: `Converter.Convert` (wazero runs `mobitool.wasm` on a temp copy) → store under key.
|
||||
3. `http.ServeContent` the `.epub` with `application/epub+zip`, byte-range supported.
|
||||
4. Client sees EPUB → existing reflow `EpubReader` renders it. Reader progress stays keyed to the **source** `fileId` (the EPUB is a derived view of the same file), so progress/annotations are unaffected.
|
||||
|
||||
## Error handling — explicit client failure contract
|
||||
Transparent raw fallback is **not** safe on its own: once the server advertises conversion and the Android in-app EPUB path opens `/read`, returning raw MOBI/AZW bytes hands the EPUB reader a non-EPUB body (Codex). So failures need a contract the client can detect:
|
||||
- On conversion failure (DRM, corrupt, timeout, oversize) the read endpoint returns a **structured signal** the Android client maps to its external-open fallback — preferred: a distinct status (e.g. `422 Unprocessable` / `415`) with an error code, **or** serve raw bytes **plus** an explicit `X-Silo-Ebook-Conversion: failed` (with the original MIME) header that the client checks. Pick one and make both sides honor it. Never silently hand raw bytes to the EPUB path.
|
||||
- `ErrDRMProtected` vs `ErrConversionFailed` are distinguished in logs + (optionally) the error code, so DRM books get the "open externally" message and transient failures can be retried.
|
||||
- Module compile/instantiate failure at startup (corrupt embed) → capability **not advertised** (gate b), kindle files serve raw as today, logged loudly.
|
||||
|
||||
## Testing
|
||||
- `internal/ebookconvert` unit tests with **small DRM-free MOBI6, AZW, and KF8/AZW3 fixtures**: `Convert` produces a valid EPUB (zip with `mimetype` first + `META-INF/container.xml` + OPF + at least one content doc), deterministic across runs; corrupt input → `ErrConversionFailed`; oversize → rejected before invocation; timeout honored (ctx cancel).
|
||||
- Cache tests: miss→convert→hit; key changes on source mtime/size and on module-version bump; singleflight collapses concurrent conversions.
|
||||
- Handler tests: enabled+kindle → `application/epub+zip` body is a valid EPUB; disabled → raw bytes + original MIME; non-kindle → unchanged; DRM/failed → graceful fallback path.
|
||||
- A build-stage check (CI) that the committed `mobitool.wasm` matches a rebuild from the pinned libmobi (or at least that it instantiates + converts a fixture), so the artifact can't silently rot.
|
||||
|
||||
## Sequencing — spike first (Codex-recommended)
|
||||
**The first PR is only the build spike + a tiny Go harness that converts fixtures** — no handler/cache/wiring until the linchpin is proven. Codex confirmed libmobi documents both deps as optional (`--with-zlib=no` → bundled `miniz.c`; `--with-libxml2=no` → internal xmlwriter) and that `mobitool.c` gates `-e` behind `USE_XMLWRITER`, not `HAVE_LIBXML2` — so the no-libxml2 wasm build is *plausible*, but the spike must **assert** `mobitool -h` exposes `-e` and a fixture actually converts. Only then design the handler/cache/wiring PR.
|
||||
|
||||
## Risks / spike-first items
|
||||
1. **libmobi+mobitool compiles to `wasm32-wasi` with the `-e` (EPUB) path working without libxml2.** The linchpin. If libxml2 turns out mandatory for `-e`, fall back to compiling libxml2 to wasm too (heavier) or assembling the EPUB in Go from libmobi's `MOBIRawml` parts.
|
||||
2. **wazero WASI preopen file IO** for the file sizes involved (tens of MB) — confirm mapping + memory/time limits.
|
||||
3. mobitool CLI surface as a wasm **command module** (argv + preopens) — confirm wazero runs it via `_start` and we capture exit status + stdout + stderr.
|
||||
|
||||
## Committed `mobitool.wasm` — guardrails (Codex)
|
||||
Commit the artifact (so normal Go builds need no wasi-sdk) **with**: the pinned libmobi source commit, the Dockerfile/build script that produced it, its SHA256, the build flags, and the LGPL license notice. Add a **CI smoke test** that instantiates the embedded module and converts a fixture. Do not gate CI on byte-for-byte rebuild reproducibility — `mobitool -v` embeds `__DATE__`/`__TIME__`, so a rebuild won't match byte-for-byte unless normalized; rely on the fixture-conversion smoke + metadata hash instead.
|
||||
|
||||
## Process gate (Codex)
|
||||
`docs/architecture/v1-scope.md` indicates v1 scope is not locked and new user-facing capabilities may need a proposal / project-tracking entry **before** a feature PR. Confirm this process step with the maintainer before opening the conversion PR. *(Decision deferred to the user.)*
|
||||
|
||||
## Out of scope
|
||||
- CBR (separate Android client-side spec).
|
||||
- DRM removal (never).
|
||||
- Converting at scan/ingest time (on-demand only).
|
||||
- Surfacing a separate "EPUB version" in the catalog (we serve transparently through `/read`).
|
||||
- Reusing the WASM converter for other formats (later, if useful).
|
||||
|
||||
## Commit / deploy rules (project)
|
||||
- silo-server changes go on a **feature branch, committed locally, PR-ready — NOT pushed** (per standing rule; server changes land via PR).
|
||||
@@ -31,6 +31,7 @@ require (
|
||||
github.com/oklog/ulid/v2 v2.1.0
|
||||
github.com/pgvector/pgvector-go v0.3.0
|
||||
github.com/pressly/goose/v3 v3.27.1
|
||||
github.com/tetratelabs/wazero v1.12.0
|
||||
github.com/wneessen/go-mail v0.7.3
|
||||
github.com/zishang520/socket.io/v2 v2.5.0
|
||||
go.n16f.net/thumbhash v1.1.0
|
||||
|
||||
@@ -197,6 +197,8 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
|
||||
github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals=
|
||||
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/tetratelabs/wazero v1.12.0 h1:DuWcpNu/FzgEXgGBDp8J1Spc+CWOvvtvVyjKlaZopYU=
|
||||
github.com/tetratelabs/wazero v1.12.0/go.mod h1:LvKtzl2RqO4gyF27BiXU+nKAjcV8f38U+kP/q2vgxh0=
|
||||
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc h1:9lRDQMhESg+zvGYmW5DyG0UqvY96Bu5QYsTLvCHdrgo=
|
||||
github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc/go.mod h1:bciPuU6GHm1iF1pBvUfxfsH0Wmnc2VbpgvbI9ZWuIRs=
|
||||
github.com/uptrace/bun v1.1.12 h1:sOjDVHxNTuM6dNGaba0wUuz7KvDE1BmNu9Gqs2gJSXQ=
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/api/handlers"
|
||||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||||
"github.com/Silo-Server/silo-server/internal/config"
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
)
|
||||
|
||||
// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB
|
||||
// conversion. Off by default; toggled via PUT /admin/settings/{key}.
|
||||
const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled"
|
||||
|
||||
// buildEbookConversion wires the in-process Kindle->EPUB converter for the read
|
||||
// handler. It compiles the embedded WASM module once at startup; if that fails
|
||||
// the feature stays off (returns nil) and the raw original is served as before.
|
||||
// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be
|
||||
// toggled without a restart yet does not hit the DB on every read.
|
||||
//
|
||||
// The Converter (a wazero runtime, ~no external resources — per-conversion
|
||||
// scratch dirs are cleaned as they go) is intentionally owned for the process
|
||||
// lifetime: it is not Close()d on shutdown because process exit reclaims it and
|
||||
// there is nothing to flush. Memory is bounded per conversion by
|
||||
// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's
|
||||
// resource notes); tune via ebookconvert.Options if the deployment is tight.
|
||||
func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion {
|
||||
if settings == nil {
|
||||
return nil
|
||||
}
|
||||
converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{})
|
||||
if err != nil {
|
||||
slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err)
|
||||
return nil
|
||||
}
|
||||
cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())})
|
||||
if err != nil {
|
||||
slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err)
|
||||
_ = converter.Close(context.Background())
|
||||
return nil
|
||||
}
|
||||
slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey)
|
||||
return &handlers.EbookConversion{
|
||||
Converter: cache,
|
||||
Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey),
|
||||
}
|
||||
}
|
||||
|
||||
// ebookFlagPredicate returns a cached boolean reader for a server setting. The
|
||||
// read path (every ebook read + the capability endpoint, which clients may
|
||||
// poll) would otherwise issue a DB query per request; a short TTL collapses
|
||||
// bursts to one query while keeping admin toggles effectively instant. On a read
|
||||
// error it keeps the last known value rather than flapping the feature off.
|
||||
func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool {
|
||||
const ttl = 3 * time.Second
|
||||
var (
|
||||
mu sync.Mutex
|
||||
value bool
|
||||
fetched time.Time
|
||||
hasVal bool
|
||||
)
|
||||
return func(ctx context.Context) bool {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if hasVal && time.Since(fetched) < ttl {
|
||||
return value
|
||||
}
|
||||
v, err := settings.Get(ctx, key)
|
||||
if err != nil {
|
||||
return value // keep last known (false until the first successful read)
|
||||
}
|
||||
value = isTruthySetting(v)
|
||||
fetched = time.Now()
|
||||
hasVal = true
|
||||
return value
|
||||
}
|
||||
}
|
||||
|
||||
// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling
|
||||
// of the transcode dir so it lives alongside other derived media.
|
||||
func ebookConversionCacheDir(cfg *config.Config) string {
|
||||
base := os.TempDir()
|
||||
if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" {
|
||||
base = filepath.Dir(cfg.Playback.TranscodeDir)
|
||||
}
|
||||
return filepath.Join(base, "silo-ebook-epub")
|
||||
}
|
||||
|
||||
func isTruthySetting(v string) bool {
|
||||
switch strings.ToLower(strings.TrimSpace(v)) {
|
||||
case "true", "1", "yes", "on", "enabled":
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"context"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// stubSettings counts Get calls so we can assert the flag predicate caches.
|
||||
type stubSettings struct {
|
||||
value string
|
||||
gets int64
|
||||
}
|
||||
|
||||
func (s *stubSettings) Get(_ context.Context, _ string) (string, error) {
|
||||
atomic.AddInt64(&s.gets, 1)
|
||||
return s.value, nil
|
||||
}
|
||||
func (s *stubSettings) Set(context.Context, string, string) error { return nil }
|
||||
func (s *stubSettings) GetAll(context.Context) (map[string]string, error) {
|
||||
return map[string]string{}, nil
|
||||
}
|
||||
|
||||
// The flag predicate must not hit the settings store on every call: the read
|
||||
// path (every ebook read + capability poll) would otherwise issue a DB query
|
||||
// per request.
|
||||
func TestEbookFlagPredicate_CachesWithinTTL(t *testing.T) {
|
||||
s := &stubSettings{value: "true"}
|
||||
pred := ebookFlagPredicate(s, "ebook.kindle_conversion_enabled")
|
||||
|
||||
for i := 0; i < 50; i++ {
|
||||
if !pred(context.Background()) {
|
||||
t.Fatalf("call %d: predicate returned false for a truthy flag", i)
|
||||
}
|
||||
}
|
||||
if got := atomic.LoadInt64(&s.gets); got > 2 {
|
||||
t.Fatalf("settings.Get called %d times across 50 predicate calls; expected it to be cached", got)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
// ConversionHeader tells the client how a kindle-family read was served.
|
||||
// "converted" => body is EPUB; "failed" => body is the raw original and the
|
||||
// client should fall back to opening it externally. Absent => no conversion
|
||||
// was attempted (feature off or non-kindle format), serve/behave as before.
|
||||
const ConversionHeader = "X-Silo-Ebook-Conversion"
|
||||
|
||||
// EbookConverter produces (and caches) a converted EPUB for a source file.
|
||||
// Implemented by *ebookconvert.Cache; an interface here for testability.
|
||||
type EbookConverter interface {
|
||||
GetOrConvert(ctx context.Context, srcPath string, key ebookconvert.SourceKey) (string, error)
|
||||
// Lookup reports a cached conversion WITHOUT converting (used by HEAD so a
|
||||
// metadata probe never kicks off a full conversion): a path on hit, a
|
||||
// non-nil error for a negatively-cached source, else a miss.
|
||||
Lookup(key ebookconvert.SourceKey) (string, error, bool)
|
||||
}
|
||||
|
||||
// EbookConversion bundles the converter with the admin-flag predicate. A nil
|
||||
// *EbookConversion (or nil Converter) means the feature is off.
|
||||
type EbookConversion struct {
|
||||
Converter EbookConverter
|
||||
// Enabled reports whether the admin flag is on for this request.
|
||||
Enabled func(ctx context.Context) bool
|
||||
}
|
||||
|
||||
func (c *EbookConversion) active(ctx context.Context) bool {
|
||||
return c != nil && c.Converter != nil && c.Enabled != nil && c.Enabled(ctx)
|
||||
}
|
||||
|
||||
// kindleReaderFormats are the Kindle-family formats the converter handles.
|
||||
var kindleReaderFormats = map[string]bool{"mobi": true, "azw": true, "azw3": true}
|
||||
|
||||
func isKindleReaderFile(file *models.MediaFile) bool {
|
||||
return file != nil && kindleReaderFormats[ebookReaderFormat(file.FilePath, file.Container)]
|
||||
}
|
||||
|
||||
// serveEbook serves an ebook file, converting Kindle-family formats to EPUB
|
||||
// when the feature is enabled. On conversion failure it serves the raw original
|
||||
// with the ConversionHeader=failed contract so the client opens it externally.
|
||||
func (h *EbookReaderHandler) serveEbook(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
|
||||
if h.Conversion.active(r.Context()) && isKindleReaderFile(file) {
|
||||
// Strong identity for both the conversion cache and the response ETag:
|
||||
// id + size + mtime (stat) + content hash + module version.
|
||||
key, statErr := ebookconvert.SourceKeyFromStat(file.ID, file.FilePath)
|
||||
if statErr != nil {
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
key.Checksum = file.FileHash
|
||||
|
||||
// HEAD must not trigger a (possibly minute-long, 1 GiB) conversion. Answer
|
||||
// from cache only; on a miss, advertise the converted representation
|
||||
// cheaply and let the GET produce the body + authoritative verdict.
|
||||
if r.Method == http.MethodHead {
|
||||
return h.serveKindleHead(w, r, file, key)
|
||||
}
|
||||
|
||||
epubPath, err := h.Conversion.Converter.GetOrConvert(r.Context(), file.FilePath, key)
|
||||
switch {
|
||||
case err == nil:
|
||||
if serveErr := serveConvertedEpub(w, r, file, epubPath, key); serveErr != nil {
|
||||
// Produced but unservable (open/stat) — honor the contract, not a 500.
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
return nil
|
||||
case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
|
||||
// Caller went away / request cancelled — not a conversion verdict.
|
||||
return err
|
||||
default:
|
||||
// DRM, corrupt, oversize, timed out, unavailable: fall back to the raw
|
||||
// original and tell the client so it can open externally.
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
}
|
||||
// Feature off or non-kindle format: serve the raw original with its native
|
||||
// MIME and no conversion header. This path deliberately omits the failed
|
||||
// contract — when conversion is disabled the capability endpoint already
|
||||
// reports `enabled:false`, so clients keep their external-open path and never
|
||||
// expect an EPUB here. (No representation flip to guard against.)
|
||||
return serveEbookInline(w, r, file)
|
||||
}
|
||||
|
||||
// serveRawKindleFallback serves the raw original with the failed-conversion
|
||||
// contract. no-store because the same URL flips raw/converted with the flag.
|
||||
func (h *EbookReaderHandler) serveRawKindleFallback(w http.ResponseWriter, r *http.Request, file *models.MediaFile) error {
|
||||
w.Header().Set(ConversionHeader, "failed")
|
||||
w.Header().Set("Cache-Control", "no-store")
|
||||
return serveEbookInline(w, r, file)
|
||||
}
|
||||
|
||||
// serveKindleHead answers a HEAD for a kindle-family file without converting:
|
||||
// - cache hit → real converted headers (and ServeContent's Content-Length);
|
||||
// - neg-cached → the failed contract (raw original headers);
|
||||
// - miss → converted headers advertised optimistically (no body, no
|
||||
// Content-Length, no conversion). A HEAD is only a hint; the GET delivers
|
||||
// the body and the authoritative DRM/failure verdict, which then populates
|
||||
// the (positive or negative) cache that subsequent HEADs read.
|
||||
func (h *EbookReaderHandler) serveKindleHead(w http.ResponseWriter, r *http.Request, file *models.MediaFile, key ebookconvert.SourceKey) error {
|
||||
path, lookupErr, ok := h.Conversion.Converter.Lookup(key)
|
||||
switch {
|
||||
case ok:
|
||||
if serveErr := serveConvertedEpub(w, r, file, path, key); serveErr != nil {
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
}
|
||||
return nil
|
||||
case lookupErr != nil:
|
||||
return h.serveRawKindleFallback(w, r, file)
|
||||
default:
|
||||
setConvertedEpubHeaders(w, file, key)
|
||||
w.WriteHeader(http.StatusOK)
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// setConvertedEpubHeaders writes the headers for a converted-EPUB response.
|
||||
// Because the same URL can return raw or converted bytes depending on the
|
||||
// flag/outcome, the ETag is derived from the exact conversion cache key (so a
|
||||
// validator can never disagree with the cache) with a revalidate policy that
|
||||
// keeps clients/proxies from serving a stale representation.
|
||||
func setConvertedEpubHeaders(w http.ResponseWriter, file *models.MediaFile, key ebookconvert.SourceKey) {
|
||||
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
|
||||
w.Header().Set("Content-Type", "application/epub+zip")
|
||||
w.Header().Set("Content-Disposition", inlineContentDisposition(name))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set(ConversionHeader, "converted")
|
||||
w.Header().Set("ETag", fmt.Sprintf("%q", "epubconv-"+key.CacheKey()))
|
||||
w.Header().Set("Cache-Control", "private, max-age=0, must-revalidate")
|
||||
}
|
||||
|
||||
// serveConvertedEpub serves the cached EPUB with EPUB headers (see
|
||||
// setConvertedEpubHeaders). For a HEAD, ServeContent emits the headers and
|
||||
// Content-Length without a body.
|
||||
func serveConvertedEpub(w http.ResponseWriter, r *http.Request, file *models.MediaFile, epubPath string, key ebookconvert.SourceKey) error {
|
||||
f, err := os.Open(epubPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("opening converted epub: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
stat, err := f.Stat()
|
||||
if err != nil {
|
||||
return fmt.Errorf("stat converted epub: %w", err)
|
||||
}
|
||||
|
||||
name := strings.TrimSuffix(filepath.Base(file.FilePath), filepath.Ext(file.FilePath)) + ".epub"
|
||||
setConvertedEpubHeaders(w, file, key)
|
||||
http.ServeContent(w, r, name, stat.ModTime(), f)
|
||||
return nil
|
||||
}
|
||||
|
||||
// ebookConversionCapability is the client-facing advertisement for the
|
||||
// Kindle->EPUB feature. Clients flip mobi/azw/azw3 to in-app reading only when
|
||||
// Enabled is true; otherwise they keep the external-open path.
|
||||
type ebookConversionCapability struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
SourceFormats []string `json:"source_formats"`
|
||||
ServedFormat string `json:"served_format"`
|
||||
Header string `json:"header"`
|
||||
HeaderOnFailed string `json:"header_failed_value"`
|
||||
}
|
||||
|
||||
// HandleConversionCapability reports whether Kindle->EPUB conversion is active
|
||||
// (admin flag on AND the converter initialized). GET /api/v1/ebooks/capability.
|
||||
func (h *EbookReaderHandler) HandleConversionCapability(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, http.StatusOK, ebookConversionCapability{
|
||||
Enabled: h.Conversion.active(r.Context()),
|
||||
SourceFormats: []string{"mobi", "azw", "azw3"},
|
||||
ServedFormat: "epub",
|
||||
Header: ConversionHeader,
|
||||
HeaderOnFailed: "failed",
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,271 @@
|
||||
package handlers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||||
"github.com/Silo-Server/silo-server/internal/models"
|
||||
)
|
||||
|
||||
type fakeConverter struct {
|
||||
epubPath string
|
||||
err error
|
||||
calls int
|
||||
|
||||
lookupPath string
|
||||
lookupErr error
|
||||
lookupOK bool
|
||||
}
|
||||
|
||||
func (f *fakeConverter) GetOrConvert(_ context.Context, _ string, _ ebookconvert.SourceKey) (string, error) {
|
||||
f.calls++
|
||||
return f.epubPath, f.err
|
||||
}
|
||||
|
||||
func (f *fakeConverter) Lookup(_ ebookconvert.SourceKey) (string, error, bool) {
|
||||
return f.lookupPath, f.lookupErr, f.lookupOK
|
||||
}
|
||||
|
||||
func writeTemp(t *testing.T, name, content string) string {
|
||||
t.Helper()
|
||||
p := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
func serveAndRecord(t *testing.T, h *EbookReaderHandler, file *models.MediaFile) *httptest.ResponseRecorder {
|
||||
t.Helper()
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook: %v", err)
|
||||
}
|
||||
return rec
|
||||
}
|
||||
|
||||
func enabledConversion(conv EbookConverter, on bool) *EbookConversion {
|
||||
return &EbookConversion{Converter: conv, Enabled: func(context.Context) bool { return on }}
|
||||
}
|
||||
|
||||
func TestServeEbook_ConvertsKindleWhenEnabled(t *testing.T) {
|
||||
epub := writeTemp(t, "out.epub", "PK\x03\x04 fake-epub-bytes")
|
||||
src := writeTemp(t, "book.azw3", "raw-azw3-source")
|
||||
conv := &fakeConverter{epubPath: epub}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3", FileHash: "abc123"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
|
||||
}
|
||||
if rec.Header().Get("ETag") == "" {
|
||||
t.Fatal("expected an ETag on converted response")
|
||||
}
|
||||
if rec.Body.String() != "PK\x03\x04 fake-epub-bytes" {
|
||||
t.Fatalf("body = %q, want the epub bytes", rec.Body.String())
|
||||
}
|
||||
if conv.calls != 1 {
|
||||
t.Fatalf("converter called %d times, want 1", conv.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_ConversionFailureFallsBackToRawWithHeader(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
|
||||
conv := &fakeConverter{err: ebookconvert.ErrDRMProtected}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 2, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get(ConversionHeader); got != "failed" {
|
||||
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
|
||||
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
|
||||
}
|
||||
if rec.Body.String() != "raw-mobi-bytes" {
|
||||
t.Fatalf("body = %q, want raw mobi bytes", rec.Body.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_DisabledServesRaw(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw-mobi-bytes")
|
||||
conv := &fakeConverter{epubPath: "should-not-be-used"}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, false)}
|
||||
|
||||
file := &models.MediaFile{ID: 3, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if got := rec.Header().Get(ConversionHeader); got != "" {
|
||||
t.Fatalf("%s = %q, want empty (no conversion attempted)", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/x-mobipocket-ebook" {
|
||||
t.Fatalf("Content-Type = %q, want raw mobi mime", got)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("converter called %d times while disabled, want 0", conv.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_NonKindleUntouched(t *testing.T) {
|
||||
raw := writeTemp(t, "book.epub", "epub-bytes")
|
||||
conv := &fakeConverter{epubPath: "should-not-be-used"}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
|
||||
file := &models.MediaFile{ID: 4, FilePath: raw, Container: "epub"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
|
||||
if rec.Header().Get(ConversionHeader) != "" {
|
||||
t.Fatal("non-kindle file must not trigger conversion")
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("converter called %d times for epub, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want epub", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_NilConversionServesRaw(t *testing.T) {
|
||||
raw := writeTemp(t, "book.mobi", "raw")
|
||||
h := &EbookReaderHandler{} // Conversion nil
|
||||
file := &models.MediaFile{ID: 5, FilePath: raw, Container: "mobi"}
|
||||
rec := serveAndRecord(t, h, file)
|
||||
if rec.Header().Get(ConversionHeader) != "" {
|
||||
t.Fatal("nil Conversion must not set the conversion header")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHandleConversionCapability(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
conv *EbookConversion
|
||||
want bool
|
||||
}{
|
||||
{"enabled", enabledConversion(&fakeConverter{}, true), true},
|
||||
{"flag-off", enabledConversion(&fakeConverter{}, false), false},
|
||||
{"not-wired", nil, false},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
h := &EbookReaderHandler{Conversion: tc.conv}
|
||||
rec := httptest.NewRecorder()
|
||||
h.HandleConversionCapability(rec, httptest.NewRequest(http.MethodGet, "/ebooks/capability", nil))
|
||||
if rec.Code != http.StatusOK {
|
||||
t.Fatalf("status = %d", rec.Code)
|
||||
}
|
||||
body := rec.Body.String()
|
||||
wantField := `"enabled":false`
|
||||
if tc.want {
|
||||
wantField = `"enabled":true`
|
||||
}
|
||||
if !strings.Contains(body, wantField) {
|
||||
t.Fatalf("body %q missing %s", body, wantField)
|
||||
}
|
||||
if !strings.Contains(body, `"served_format":"epub"`) || !strings.Contains(body, "mobi") {
|
||||
t.Fatalf("capability body incomplete: %s", body)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD on a cache miss must NOT trigger a (possibly minute-long) conversion;
|
||||
// it advertises the converted representation cheaply and lets the GET do the work.
|
||||
func TestServeEbook_HeadDoesNotConvertOnMiss(t *testing.T) {
|
||||
src := writeTemp(t, "book.azw3", "raw-azw3-source")
|
||||
conv := &fakeConverter{lookupOK: false} // cache miss
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 1, FilePath: src, Container: "azw3"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a cache miss triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted (advertised on HEAD miss)", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
if rec.Header().Get("ETag") == "" {
|
||||
t.Fatal("HEAD miss should still carry the conversion ETag")
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD that hits the cache serves the converted headers from the cached file,
|
||||
// without converting.
|
||||
func TestServeEbook_HeadServesCachedConverted(t *testing.T) {
|
||||
epub := writeTemp(t, "out.epub", "PK\x03\x04 cached-epub")
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{lookupPath: epub, lookupOK: true}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 2, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a cache hit triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "converted" {
|
||||
t.Fatalf("%s = %q, want converted", ConversionHeader, got)
|
||||
}
|
||||
if got := rec.Header().Get("Content-Type"); got != "application/epub+zip" {
|
||||
t.Fatalf("Content-Type = %q, want application/epub+zip", got)
|
||||
}
|
||||
}
|
||||
|
||||
// A HEAD on a negatively-cached (DRM/failed) source reports the failed contract
|
||||
// without converting.
|
||||
func TestServeEbook_HeadDRMReturnsFailed(t *testing.T) {
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{lookupErr: ebookconvert.ErrDRMProtected}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 3, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodHead, "/read", nil)
|
||||
if err := h.serveEbook(rec, req, file); err != nil {
|
||||
t.Fatalf("serveEbook HEAD: %v", err)
|
||||
}
|
||||
if conv.calls != 0 {
|
||||
t.Fatalf("HEAD on a negatively-cached source triggered %d conversions, want 0", conv.calls)
|
||||
}
|
||||
if got := rec.Header().Get(ConversionHeader); got != "failed" {
|
||||
t.Fatalf("%s = %q, want failed", ConversionHeader, got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestServeEbook_ContextCancelPropagates(t *testing.T) {
|
||||
src := writeTemp(t, "book.mobi", "raw-mobi-source")
|
||||
conv := &fakeConverter{err: context.Canceled}
|
||||
h := &EbookReaderHandler{Conversion: enabledConversion(conv, true)}
|
||||
file := &models.MediaFile{ID: 6, FilePath: src, Container: "mobi"}
|
||||
|
||||
rec := httptest.NewRecorder()
|
||||
req := httptest.NewRequest(http.MethodGet, "/read", nil)
|
||||
err := h.serveEbook(rec, req, file)
|
||||
if err == nil {
|
||||
t.Fatal("expected context cancellation to propagate, not fall back")
|
||||
}
|
||||
if rec.Header().Get(ConversionHeader) == "failed" {
|
||||
t.Fatal("cancellation must not be reported as a conversion failure")
|
||||
}
|
||||
}
|
||||
@@ -96,6 +96,9 @@ type EbookReaderHandler struct {
|
||||
ProgressStore EbookReaderProgressStore
|
||||
ConfigStore EbookReaderConfigStore
|
||||
AnnotationStore EbookReaderAnnotationStore
|
||||
// Conversion, when set and enabled, transparently converts Kindle-family
|
||||
// files to EPUB on read. Nil means the feature is off.
|
||||
Conversion *EbookConversion
|
||||
}
|
||||
|
||||
func NewEbookReaderHandler(authorizer *MediaFileAuthorizer) *EbookReaderHandler {
|
||||
@@ -130,7 +133,7 @@ func (h *EbookReaderHandler) HandleReadFile(w http.ResponseWriter, r *http.Reque
|
||||
return
|
||||
}
|
||||
|
||||
if err := serveEbookInline(w, r, file); err != nil {
|
||||
if err := h.serveEbook(w, r, file); err != nil {
|
||||
if errors.Is(err, catalog.ErrItemNotFound) {
|
||||
writeError(w, http.StatusNotFound, "not_found", "Ebook file not found")
|
||||
return
|
||||
|
||||
@@ -515,6 +515,9 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if ebookAnnotationStore != nil {
|
||||
ebookReaderHandler.AnnotationStore = ebookAnnotationStore
|
||||
}
|
||||
if conv := buildEbookConversion(deps, settingsRepo); conv != nil {
|
||||
ebookReaderHandler.Conversion = conv
|
||||
}
|
||||
}
|
||||
catalogResourceHandler = handlers.NewCatalogResourceHandler(itemsHandler)
|
||||
catalogHandler = handlers.NewCatalogHandler(
|
||||
@@ -1993,6 +1996,7 @@ func NewRouter(deps Dependencies) chi.Router {
|
||||
if ebookReaderHandler != nil {
|
||||
r.Route("/ebooks", func(r chi.Router) {
|
||||
r.Use(apimw.RequireProfile)
|
||||
r.Get("/capability", ebookReaderHandler.HandleConversionCapability)
|
||||
r.Get("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
|
||||
r.Head("/{content_id}/files/{file_id}/read", ebookReaderHandler.HandleReadFile)
|
||||
r.Get("/{content_id}/progress", ebookReaderHandler.HandleGetProgress)
|
||||
|
||||
@@ -0,0 +1,284 @@
|
||||
package ebookconvert
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"golang.org/x/sync/singleflight"
|
||||
)
|
||||
|
||||
// moduleVersion is a short fingerprint of the embedded wasm, mixed into every
|
||||
// cache key so bumping mobitool.wasm transparently invalidates old conversions.
|
||||
var moduleVersion = func() string {
|
||||
sum := sha256.Sum256(mobitoolWasm)
|
||||
return hex.EncodeToString(sum[:])[:16]
|
||||
}()
|
||||
|
||||
// ModuleVersion returns the embedded converter's fingerprint. Callers can mix
|
||||
// it into HTTP ETags so a converter bump invalidates client/proxy caches.
|
||||
func ModuleVersion() string { return moduleVersion }
|
||||
|
||||
// SourceKey identifies a source file for caching. Built by the caller from the
|
||||
// catalog/file metadata. Size+ModTimeNano are cheap and computed on every read;
|
||||
// Checksum (the scanner's content hash, if available) hardens against a replace
|
||||
// that preserves size+mtime — set it when you have it.
|
||||
type SourceKey struct {
|
||||
FileID int
|
||||
Size int64
|
||||
ModTimeNano int64
|
||||
Checksum string // optional; scanner checksum
|
||||
}
|
||||
|
||||
func (k SourceKey) hash() string {
|
||||
h := sha256.New()
|
||||
fmt.Fprintf(h, "v1|mod=%s|id=%d|sz=%d|mt=%d|ck=%s",
|
||||
moduleVersion, k.FileID, k.Size, k.ModTimeNano, k.Checksum)
|
||||
return hex.EncodeToString(h.Sum(nil))
|
||||
}
|
||||
|
||||
// CacheKey returns the exact identity used to cache this source's conversion
|
||||
// (source identity + converter module version). Use it to derive an HTTP ETag
|
||||
// so the validator and the cache entry can never disagree.
|
||||
func (k SourceKey) CacheKey() string { return k.hash() }
|
||||
|
||||
// CacheOptions configures a Cache.
|
||||
type CacheOptions struct {
|
||||
// Dir is the on-disk cache root for converted EPUBs (required).
|
||||
Dir string
|
||||
// MaxBytes bounds total cached EPUB size; oldest are evicted past it.
|
||||
// Zero uses DefaultCacheMaxBytes.
|
||||
MaxBytes int64
|
||||
// NegativeTTL is how long a DRM/failed result is remembered to avoid
|
||||
// reconverting a known-bad source. Zero uses DefaultNegativeTTL.
|
||||
NegativeTTL time.Duration
|
||||
}
|
||||
|
||||
const (
|
||||
DefaultCacheMaxBytes = 2 << 30 // 2 GiB
|
||||
DefaultNegativeTTL = 6 * time.Hour
|
||||
)
|
||||
|
||||
// Cache wraps a Converter with an on-disk, size-bounded, singleflighted cache
|
||||
// and an in-memory negative cache for DRM/failed sources.
|
||||
type Cache struct {
|
||||
conv *Converter
|
||||
opts CacheOptions
|
||||
|
||||
group singleflight.Group
|
||||
|
||||
mu sync.Mutex
|
||||
neg map[string]negEntry // key hash -> negative result
|
||||
}
|
||||
|
||||
type negEntry struct {
|
||||
err error
|
||||
expiry time.Time
|
||||
}
|
||||
|
||||
// NewCache creates the cache dir and returns a ready Cache.
|
||||
func NewCache(conv *Converter, opts CacheOptions) (*Cache, error) {
|
||||
if conv == nil {
|
||||
return nil, ErrUnavailable
|
||||
}
|
||||
if opts.Dir == "" {
|
||||
return nil, errors.New("ebookconvert: cache Dir is required")
|
||||
}
|
||||
if opts.MaxBytes <= 0 {
|
||||
opts.MaxBytes = DefaultCacheMaxBytes
|
||||
}
|
||||
if opts.NegativeTTL <= 0 {
|
||||
opts.NegativeTTL = DefaultNegativeTTL
|
||||
}
|
||||
if err := os.MkdirAll(opts.Dir, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("ebookconvert: create cache dir: %w", err)
|
||||
}
|
||||
return &Cache{conv: conv, opts: opts, neg: make(map[string]negEntry)}, nil
|
||||
}
|
||||
|
||||
// GetOrConvert returns the path to a cached EPUB for srcPath/key, converting on
|
||||
// miss. Concurrent calls for the same key collapse to a single conversion.
|
||||
// Returns ErrDRMProtected / ErrConversionFailed (also served from the negative
|
||||
// cache), ErrConversionTimedOut, ErrSourceTooLarge, or the caller's context
|
||||
// error if it cancels while waiting.
|
||||
func (c *Cache) GetOrConvert(ctx context.Context, srcPath string, key SourceKey) (string, error) {
|
||||
kh := key.hash()
|
||||
dst := filepath.Join(c.opts.Dir, kh+".epub")
|
||||
|
||||
// Fast path: already converted. Refresh mtime so the mtime-ordered budget
|
||||
// eviction behaves as a real LRU — recently-read entries sort newest and
|
||||
// survive eviction.
|
||||
if fi, err := os.Stat(dst); err == nil && fi.Size() > 0 {
|
||||
now := time.Now()
|
||||
_ = os.Chtimes(dst, now, now)
|
||||
return dst, nil
|
||||
}
|
||||
|
||||
// Negative cache: known DRM/deterministically-bad source.
|
||||
if err := c.negativeLookup(kh); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// Singleflight via DoChan with a *detached* context: coalesced callers share
|
||||
// one conversion, so a single caller canceling its request must not abort
|
||||
// the shared work for the others (golang.org/x/sync/singleflight.Do has no
|
||||
// per-caller context, and binding it to the first caller would propagate that
|
||||
// caller's cancellation to all). The conversion stays bounded by the
|
||||
// Converter's own timeout + concurrency semaphore; the caller's context only
|
||||
// governs how long *this* caller waits (the select below). A caller that
|
||||
// gives up still leaves the conversion running to populate the cache.
|
||||
ch := c.group.DoChan(kh, func() (interface{}, error) {
|
||||
// Re-check after acquiring the singleflight slot (another caller may
|
||||
// have just produced it).
|
||||
if fi, statErr := os.Stat(dst); statErr == nil && fi.Size() > 0 {
|
||||
return dst, nil
|
||||
}
|
||||
// Convert to a temp file in the cache dir, then atomically rename in.
|
||||
tmp, tmpErr := os.CreateTemp(c.opts.Dir, "converting-*.epub")
|
||||
if tmpErr != nil {
|
||||
return "", fmt.Errorf("%w: cache temp: %v", ErrConversionFailed, tmpErr)
|
||||
}
|
||||
tmpPath := tmp.Name()
|
||||
_ = tmp.Close()
|
||||
_ = os.Remove(tmpPath) // Convert recreates it
|
||||
|
||||
if convErr := c.conv.Convert(context.WithoutCancel(ctx), srcPath, tmpPath); convErr != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
c.remember(kh, convErr)
|
||||
return "", convErr
|
||||
}
|
||||
if renErr := os.Rename(tmpPath, dst); renErr != nil {
|
||||
_ = os.Remove(tmpPath)
|
||||
return "", fmt.Errorf("%w: cache rename: %v", ErrConversionFailed, renErr)
|
||||
}
|
||||
c.enforceBudget(dst)
|
||||
return dst, nil
|
||||
})
|
||||
|
||||
select {
|
||||
case res := <-ch:
|
||||
if res.Err != nil {
|
||||
return "", res.Err
|
||||
}
|
||||
return res.Val.(string), nil
|
||||
case <-ctx.Done():
|
||||
return "", ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
// Lookup returns a cached conversion result WITHOUT converting on miss. It lets
|
||||
// cheap callers (e.g. HEAD requests) avoid kicking off a full conversion:
|
||||
// - ok == true, err == nil: a ready EPUB exists at path.
|
||||
// - err != nil: the source is negatively cached (DRM/failed); path is empty.
|
||||
// - ok == false, err == nil: cache miss — caller must decide whether to convert.
|
||||
func (c *Cache) Lookup(key SourceKey) (path string, err error, ok bool) {
|
||||
kh := key.hash()
|
||||
dst := filepath.Join(c.opts.Dir, kh+".epub")
|
||||
if fi, statErr := os.Stat(dst); statErr == nil && fi.Size() > 0 {
|
||||
now := time.Now()
|
||||
_ = os.Chtimes(dst, now, now) // a HEAD hit counts as access for LRU
|
||||
return dst, nil, true
|
||||
}
|
||||
if negErr := c.negativeLookup(kh); negErr != nil {
|
||||
return "", negErr, false
|
||||
}
|
||||
return "", nil, false
|
||||
}
|
||||
|
||||
func (c *Cache) negativeLookup(kh string) error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
e, ok := c.neg[kh]
|
||||
if !ok {
|
||||
return nil
|
||||
}
|
||||
if time.Now().After(e.expiry) {
|
||||
delete(c.neg, kh)
|
||||
return nil
|
||||
}
|
||||
return e.err
|
||||
}
|
||||
|
||||
// remember negatively caches only *deterministic* bad outcomes — DRM
|
||||
// (ErrDRMProtected) and conversion failures (ErrConversionFailed: corrupt
|
||||
// source, unconvertible layout, oversize output) — so repeat reads of a
|
||||
// known-bad file skip reconversion for NegativeTTL. Transient outcomes
|
||||
// (ErrConversionTimedOut, context cancellation/deadline, ErrSourceTooLarge) are
|
||||
// deliberately NOT cached so the next read can retry — a one-off timeout under
|
||||
// load must not wedge a convertible book onto the raw-fallback path.
|
||||
func (c *Cache) remember(kh string, err error) {
|
||||
if !errors.Is(err, ErrDRMProtected) && !errors.Is(err, ErrConversionFailed) {
|
||||
return
|
||||
}
|
||||
c.mu.Lock()
|
||||
c.neg[kh] = negEntry{err: err, expiry: time.Now().Add(c.opts.NegativeTTL)}
|
||||
c.mu.Unlock()
|
||||
}
|
||||
|
||||
// enforceBudget evicts the oldest (by mtime) finished cache entries until the
|
||||
// total is within MaxBytes. It never evicts keep (the entry the caller is about
|
||||
// to return) and ignores in-flight "converting-*" temp files, since deleting
|
||||
// either would hand back a vanished path or corrupt a concurrent conversion.
|
||||
// Best-effort; logs nothing (caller has no logger here).
|
||||
func (c *Cache) enforceBudget(keep string) {
|
||||
entries, err := os.ReadDir(c.opts.Dir)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
type item struct {
|
||||
path string
|
||||
size int64
|
||||
mod time.Time
|
||||
}
|
||||
var items []item
|
||||
var total int64
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if e.IsDir() || filepath.Ext(name) != ".epub" || strings.HasPrefix(name, "converting-") {
|
||||
continue // skip dirs, non-epubs, and other conversions' in-flight temps
|
||||
}
|
||||
fi, statErr := e.Info()
|
||||
if statErr != nil {
|
||||
continue
|
||||
}
|
||||
items = append(items, item{filepath.Join(c.opts.Dir, name), fi.Size(), fi.ModTime()})
|
||||
total += fi.Size()
|
||||
}
|
||||
if total <= c.opts.MaxBytes {
|
||||
return
|
||||
}
|
||||
sort.Slice(items, func(i, j int) bool { return items[i].mod.Before(items[j].mod) })
|
||||
for _, it := range items {
|
||||
if total <= c.opts.MaxBytes {
|
||||
break
|
||||
}
|
||||
if it.path == keep {
|
||||
continue // never evict the entry we're about to return
|
||||
}
|
||||
if os.Remove(it.path) == nil {
|
||||
total -= it.size
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// SourceKeyFromStat builds a SourceKey from a file path + fileID, reading
|
||||
// size/mtime via stat. Checksum is left empty (pass one explicitly if known).
|
||||
func SourceKeyFromStat(fileID int, path string) (SourceKey, error) {
|
||||
fi, err := os.Stat(path)
|
||||
if err != nil {
|
||||
return SourceKey{}, err
|
||||
}
|
||||
return SourceKey{FileID: fileID, Size: fi.Size(), ModTimeNano: fi.ModTime().UnixNano()}, nil
|
||||
}
|
||||
|
||||
// for tests / diagnostics
|
||||
func (k SourceKey) String() string { return strconv.Itoa(k.FileID) + ":" + k.hash()[:8] }
|
||||
@@ -0,0 +1,297 @@
|
||||
package ebookconvert
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func newTestCache(t *testing.T, opts CacheOptions) *Cache {
|
||||
t.Helper()
|
||||
conv := newTestConverter(t)
|
||||
if opts.Dir == "" {
|
||||
opts.Dir = t.TempDir()
|
||||
}
|
||||
c, err := NewCache(conv, opts)
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache: %v", err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func keyFor(t *testing.T, fileID int, path string) SourceKey {
|
||||
t.Helper()
|
||||
k, err := SourceKeyFromStat(fileID, path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return k
|
||||
}
|
||||
|
||||
func TestCache_MissThenHit(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
key := keyFor(t, 1, src)
|
||||
|
||||
p1, err := c.GetOrConvert(context.Background(), src, key)
|
||||
if err != nil {
|
||||
t.Fatalf("first GetOrConvert: %v", err)
|
||||
}
|
||||
assertValidEpub(t, p1)
|
||||
fi1, _ := os.Stat(p1)
|
||||
|
||||
p2, err := c.GetOrConvert(context.Background(), src, key)
|
||||
if err != nil {
|
||||
t.Fatalf("second GetOrConvert: %v", err)
|
||||
}
|
||||
if p1 != p2 {
|
||||
t.Fatalf("hit returned different path: %s vs %s", p1, p2)
|
||||
}
|
||||
fi2, _ := os.Stat(p2)
|
||||
// A hit refreshes mtime for LRU but must NOT reconvert: the cached entry is
|
||||
// the same underlying file, not a freshly converted+renamed replacement.
|
||||
if !os.SameFile(fi1, fi2) {
|
||||
t.Fatal("cache hit reconverted the file (replaced the cached entry)")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_KeyChangesOnModTime(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
k1 := keyFor(t, 1, src)
|
||||
k2 := k1
|
||||
k2.ModTimeNano = k1.ModTimeNano + 1 // simulate a re-scanned/replaced file
|
||||
if k1.hash() == k2.hash() {
|
||||
t.Fatal("expected different cache keys for different mtime")
|
||||
}
|
||||
p1, err := c.GetOrConvert(context.Background(), src, k1)
|
||||
if err != nil {
|
||||
t.Fatalf("convert k1: %v", err)
|
||||
}
|
||||
p2, err := c.GetOrConvert(context.Background(), src, k2)
|
||||
if err != nil {
|
||||
t.Fatalf("convert k2: %v", err)
|
||||
}
|
||||
if p1 == p2 {
|
||||
t.Fatal("different keys must map to different cache files")
|
||||
}
|
||||
}
|
||||
|
||||
// A transient timeout is NOT negatively cached: it would otherwise wedge a
|
||||
// slow-but-convertible book onto the raw-fallback path for the whole TTL.
|
||||
func TestCache_TimeoutNotNegativelyCached(t *testing.T) {
|
||||
conv, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond})
|
||||
if err != nil {
|
||||
t.Fatalf("NewConverter: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = conv.Close(context.Background()) })
|
||||
c, err := NewCache(conv, CacheOptions{Dir: t.TempDir()})
|
||||
if err != nil {
|
||||
t.Fatalf("NewCache: %v", err)
|
||||
}
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
key := keyFor(t, 1, src)
|
||||
|
||||
_, err = c.GetOrConvert(context.Background(), src, key)
|
||||
if !errors.Is(err, ErrConversionTimedOut) {
|
||||
t.Fatalf("got %v, want ErrConversionTimedOut", err)
|
||||
}
|
||||
c.mu.Lock()
|
||||
_, cached := c.neg[key.hash()]
|
||||
c.mu.Unlock()
|
||||
if cached {
|
||||
t.Fatal("a transient timeout must not be negatively cached")
|
||||
}
|
||||
}
|
||||
|
||||
// Eviction must never delete the EPUB it is about to hand back, even when the
|
||||
// configured budget is smaller than a single converted file.
|
||||
func TestCache_EvictionKeepsJustConverted(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{Dir: t.TempDir(), MaxBytes: 1})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
p, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src))
|
||||
if err != nil {
|
||||
t.Fatalf("convert: %v", err)
|
||||
}
|
||||
if fi, statErr := os.Stat(p); statErr != nil || fi.Size() == 0 {
|
||||
t.Fatalf("returned path must exist and be non-empty (eviction deleted it): stat=%v", statErr)
|
||||
}
|
||||
}
|
||||
|
||||
// Eviction must ignore other conversions' in-flight temp files; deleting one
|
||||
// would corrupt a concurrent conversion mid-flight.
|
||||
func TestCache_EvictionSkipsInFlightTemps(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
c := newTestCache(t, CacheOptions{Dir: dir, MaxBytes: 1})
|
||||
tmp := filepath.Join(dir, "converting-inflight.epub")
|
||||
if err := os.WriteFile(tmp, make([]byte, 4096), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
old := time.Now().Add(-time.Hour) // make the temp the "oldest" candidate
|
||||
_ = os.Chtimes(tmp, old, old)
|
||||
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
if _, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src)); err != nil {
|
||||
t.Fatalf("convert: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(tmp); err != nil {
|
||||
t.Fatalf("eviction deleted an in-flight temp file: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A cache hit refreshes mtime so the mtime-ordered budget eviction behaves as a
|
||||
// real LRU (most-recently-read entries survive).
|
||||
func TestCache_HitTouchesModTimeForLRU(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
key := keyFor(t, 1, src)
|
||||
p, err := c.GetOrConvert(context.Background(), src, key)
|
||||
if err != nil {
|
||||
t.Fatalf("convert: %v", err)
|
||||
}
|
||||
old := time.Now().Add(-time.Hour)
|
||||
_ = os.Chtimes(p, old, old)
|
||||
|
||||
if _, err := c.GetOrConvert(context.Background(), src, key); err != nil {
|
||||
t.Fatalf("hit: %v", err)
|
||||
}
|
||||
fi, _ := os.Stat(p)
|
||||
if time.Since(fi.ModTime()) > time.Minute {
|
||||
t.Fatalf("cache hit did not refresh mtime for LRU; mtime is %v old", time.Since(fi.ModTime()))
|
||||
}
|
||||
}
|
||||
|
||||
// A single caller abandoning its request (cancellation) must not abort the
|
||||
// shared conversion for everyone else: the detached work still completes and
|
||||
// populates the cache.
|
||||
func TestCache_CallerCancelDoesNotAbortSharedWork(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
key := keyFor(t, 1, src)
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // caller bails before the conversion can finish
|
||||
_, _ = c.GetOrConvert(ctx, src, key)
|
||||
|
||||
dst := filepath.Join(c.opts.Dir, key.hash()+".epub")
|
||||
deadline := time.Now().Add(10 * time.Second)
|
||||
for time.Now().Before(deadline) {
|
||||
if fi, err := os.Stat(dst); err == nil && fi.Size() > 0 {
|
||||
return // shared work completed despite the caller canceling — good
|
||||
}
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
}
|
||||
t.Fatal("detached conversion did not populate the cache after the caller canceled")
|
||||
}
|
||||
|
||||
func TestCache_NegativeCachesDRM(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-drm-v1.mobi")
|
||||
key := keyFor(t, 7, src)
|
||||
|
||||
err1 := mustConvertErr(t, c, src, key)
|
||||
if !errors.Is(err1, ErrDRMProtected) {
|
||||
t.Fatalf("got %v, want ErrDRMProtected", err1)
|
||||
}
|
||||
// Second call should be served from the negative cache (still DRM error).
|
||||
err2 := mustConvertErr(t, c, src, key)
|
||||
if !errors.Is(err2, ErrDRMProtected) {
|
||||
t.Fatalf("negative-cached call: got %v, want ErrDRMProtected", err2)
|
||||
}
|
||||
c.mu.Lock()
|
||||
_, cached := c.neg[key.hash()]
|
||||
c.mu.Unlock()
|
||||
if !cached {
|
||||
t.Fatal("DRM result was not negatively cached")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_Singleflight(t *testing.T) {
|
||||
c := newTestCache(t, CacheOptions{})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
key := keyFor(t, 3, src)
|
||||
|
||||
const n = 8
|
||||
var wg sync.WaitGroup
|
||||
var firstPath atomic.Value
|
||||
errs := make(chan error, n)
|
||||
for i := 0; i < n; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
p, err := c.GetOrConvert(context.Background(), src, key)
|
||||
if err != nil {
|
||||
errs <- err
|
||||
return
|
||||
}
|
||||
firstPath.Store(p)
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
close(errs)
|
||||
for err := range errs {
|
||||
t.Fatalf("concurrent GetOrConvert: %v", err)
|
||||
}
|
||||
if p, _ := firstPath.Load().(string); p == "" {
|
||||
t.Fatal("no path produced")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCache_BudgetEviction(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// Budget fits one converted EPUB (~2.4 KiB) but not two, so converting a
|
||||
// second entry evicts the older one — while never evicting the entry just
|
||||
// handed back.
|
||||
const budget = 3000
|
||||
c := newTestCache(t, CacheOptions{Dir: dir, MaxBytes: budget})
|
||||
src := filepath.Join("testdata", "sample-ncx.mobi")
|
||||
|
||||
p1, err := c.GetOrConvert(context.Background(), src, keyFor(t, 1, src))
|
||||
if err != nil {
|
||||
t.Fatalf("convert 1: %v", err)
|
||||
}
|
||||
// Force the first entry to be the oldest (the eviction victim).
|
||||
old := time.Now().Add(-time.Hour)
|
||||
_ = os.Chtimes(p1, old, old)
|
||||
|
||||
k2 := keyFor(t, 2, src)
|
||||
k2.FileID = 2
|
||||
p2, err := c.GetOrConvert(context.Background(), src, k2)
|
||||
if err != nil {
|
||||
t.Fatalf("convert 2: %v", err)
|
||||
}
|
||||
|
||||
if _, err := os.Stat(p2); err != nil {
|
||||
t.Fatalf("just-converted entry must survive eviction: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(p1); !os.IsNotExist(err) {
|
||||
t.Fatalf("older entry should have been evicted, stat err = %v", err)
|
||||
}
|
||||
|
||||
epubs, _ := filepath.Glob(filepath.Join(dir, "*.epub"))
|
||||
var total int64
|
||||
for _, e := range epubs {
|
||||
fi, _ := os.Stat(e)
|
||||
total += fi.Size()
|
||||
}
|
||||
if total > budget {
|
||||
t.Fatalf("budget not enforced: total=%d > %d (%d files)", total, budget, len(epubs))
|
||||
}
|
||||
}
|
||||
|
||||
func TestModuleVersion_Stable(t *testing.T) {
|
||||
if len(moduleVersion) != 16 {
|
||||
t.Fatalf("moduleVersion len = %d, want 16", len(moduleVersion))
|
||||
}
|
||||
}
|
||||
|
||||
func mustConvertErr(t *testing.T, c *Cache, src string, key SourceKey) error {
|
||||
t.Helper()
|
||||
_, err := c.GetOrConvert(context.Background(), src, key)
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,468 @@
|
||||
package ebookconvert
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"bytes"
|
||||
"context"
|
||||
_ "embed"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/tetratelabs/wazero"
|
||||
"github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1"
|
||||
"github.com/tetratelabs/wazero/sys"
|
||||
)
|
||||
|
||||
// mobitoolWasm is libmobi's mobitool compiled to wasm32-wasi.
|
||||
// Built by tools/mobitool-wasm/Dockerfile; provenance in mobitool.wasm.sha256.
|
||||
//
|
||||
//go:embed mobitool.wasm
|
||||
var mobitoolWasm []byte
|
||||
|
||||
// Guest layout. Input is mounted READ-ONLY (confined fs.FS, no `..` escape);
|
||||
// output is a dedicated writable dir. NOTE: a writable wazero dir mount is not
|
||||
// a hard jail (`..` traversal is possible within the server user's reach), so
|
||||
// the WASM sandbox is a *memory-safety* boundary — filesystem isolation relies
|
||||
// on running the server as a constrained, non-root user/container. See the
|
||||
// design doc's "Filesystem sandboxing" section.
|
||||
const (
|
||||
guestInDir = "/in"
|
||||
guestOutDir = "/out"
|
||||
guestInputName = "in.mobi" // mobitool sniffs content; extension is cosmetic
|
||||
guestEpubName = "in.epub" // mobitool derives this from the input basename
|
||||
)
|
||||
|
||||
// mobitool diagnostic lines that mean the source is encrypted/DRM'd. Matched as
|
||||
// substrings against individual output lines (not the whole blob) so book
|
||||
// metadata/text containing "DRM" cannot false-positive. "Document is encrypted"
|
||||
// is distinct from mobitool's "Document is not encrypted, ignoring PID/serial".
|
||||
var drmMarkers = []string{
|
||||
"Document is encrypted",
|
||||
"DRM key not found",
|
||||
"Invalid DRM pid",
|
||||
"DRM expired",
|
||||
"DRM support not included",
|
||||
}
|
||||
|
||||
// Print Replica = fixed-layout (image) Kindle book; mobitool ignores -e and
|
||||
// produces no EPUB. Detected for a clear failure message.
|
||||
const printReplicaMarker = "Print Replica"
|
||||
|
||||
// Options configures a Converter. Zero values fall back to sane defaults.
|
||||
type Options struct {
|
||||
MaxSourceBytes int64 // reject larger sources before invoking the converter
|
||||
MaxOutputBytes int64 // reject a produced EPUB larger than this
|
||||
Timeout time.Duration // bounds a single conversion
|
||||
MaxConcurrent int // caps simultaneous conversions
|
||||
MaxMemoryPages uint32 // wasm linear-memory cap (64 KiB/page)
|
||||
MaxLogBytes int // cap on captured stdout+stderr each
|
||||
WorkRoot string // parent for per-conversion scratch dirs ("" = os.TempDir)
|
||||
}
|
||||
|
||||
const (
|
||||
DefaultMaxSourceBytes = 256 << 20 // 256 MiB
|
||||
DefaultMaxOutputBytes = 512 << 20 // 512 MiB
|
||||
DefaultTimeout = 60 * time.Second
|
||||
DefaultConcurrency = 2
|
||||
DefaultMaxMemoryPages = 16384 // 1 GiB
|
||||
DefaultMaxLogBytes = 64 << 10
|
||||
)
|
||||
|
||||
func (o *Options) applyDefaults() {
|
||||
if o.MaxSourceBytes <= 0 {
|
||||
o.MaxSourceBytes = DefaultMaxSourceBytes
|
||||
}
|
||||
if o.MaxOutputBytes <= 0 {
|
||||
o.MaxOutputBytes = DefaultMaxOutputBytes
|
||||
}
|
||||
if o.Timeout <= 0 {
|
||||
o.Timeout = DefaultTimeout
|
||||
}
|
||||
if o.MaxConcurrent <= 0 {
|
||||
o.MaxConcurrent = DefaultConcurrency
|
||||
}
|
||||
if o.MaxMemoryPages == 0 {
|
||||
o.MaxMemoryPages = DefaultMaxMemoryPages
|
||||
}
|
||||
if o.MaxLogBytes <= 0 {
|
||||
o.MaxLogBytes = DefaultMaxLogBytes
|
||||
}
|
||||
}
|
||||
|
||||
// Converter converts Kindle-family ebooks to EPUB in-process via wazero.
|
||||
// Safe for concurrent use. The WASM module is compiled once; each conversion
|
||||
// instantiates a fresh, isolated module.
|
||||
type Converter struct {
|
||||
runtime wazero.Runtime
|
||||
compiled wazero.CompiledModule
|
||||
opts Options
|
||||
sem chan struct{}
|
||||
closed atomic.Bool
|
||||
}
|
||||
|
||||
// NewConverter compiles the embedded module and returns a ready Converter.
|
||||
// On failure it returns ErrUnavailable; callers must not advertise the
|
||||
// conversion capability when this fails.
|
||||
func NewConverter(ctx context.Context, opts Options) (*Converter, error) {
|
||||
opts.applyDefaults()
|
||||
|
||||
cfg := wazero.NewRuntimeConfig().
|
||||
WithCloseOnContextDone(true).
|
||||
WithMemoryLimitPages(opts.MaxMemoryPages)
|
||||
r := wazero.NewRuntimeWithConfig(ctx, cfg)
|
||||
if _, err := wasi_snapshot_preview1.Instantiate(ctx, r); err != nil {
|
||||
_ = r.Close(ctx)
|
||||
return nil, fmt.Errorf("%w: instantiate wasi: %v", ErrUnavailable, err)
|
||||
}
|
||||
compiled, err := r.CompileModule(ctx, mobitoolWasm)
|
||||
if err != nil {
|
||||
_ = r.Close(ctx)
|
||||
return nil, fmt.Errorf("%w: compile module: %v", ErrUnavailable, err)
|
||||
}
|
||||
return &Converter{
|
||||
runtime: r,
|
||||
compiled: compiled,
|
||||
opts: opts,
|
||||
sem: make(chan struct{}, opts.MaxConcurrent),
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Close releases the runtime. Convert returns ErrUnavailable afterward.
|
||||
func (c *Converter) Close(ctx context.Context) error {
|
||||
if c == nil || c.runtime == nil {
|
||||
return nil
|
||||
}
|
||||
c.closed.Store(true)
|
||||
return c.runtime.Close(ctx)
|
||||
}
|
||||
|
||||
// Convert reads the MOBI/AZW/AZW3 at srcPath and writes a validated EPUB to
|
||||
// dstPath. Returns ErrDRMProtected, ErrConversionFailed (deterministic),
|
||||
// ErrConversionTimedOut (transient — the per-call timeout fired),
|
||||
// ErrSourceTooLarge, ErrUnavailable, or the caller's context error on
|
||||
// cancellation/deadline.
|
||||
func (c *Converter) Convert(ctx context.Context, srcPath, dstPath string) error {
|
||||
if c.closed.Load() {
|
||||
return ErrUnavailable
|
||||
}
|
||||
info, statErr := os.Stat(srcPath)
|
||||
if statErr != nil {
|
||||
return fmt.Errorf("%w: stat source: %v", ErrConversionFailed, statErr)
|
||||
}
|
||||
if info.Size() > c.opts.MaxSourceBytes {
|
||||
return ErrSourceTooLarge
|
||||
}
|
||||
|
||||
select {
|
||||
case c.sem <- struct{}{}:
|
||||
defer func() { <-c.sem }()
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
|
||||
// Per-conversion scratch: in/ (read-only mount) and out/ (writable mount).
|
||||
scratch, err := os.MkdirTemp(c.opts.WorkRoot, "ebookconvert-")
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: scratch: %v", ErrConversionFailed, err)
|
||||
}
|
||||
defer os.RemoveAll(scratch)
|
||||
|
||||
inDir := filepath.Join(scratch, "in")
|
||||
outDir := filepath.Join(scratch, "out")
|
||||
if err := os.MkdirAll(inDir, 0o755); err != nil {
|
||||
return fmt.Errorf("%w: in dir: %v", ErrConversionFailed, err)
|
||||
}
|
||||
if err := os.MkdirAll(outDir, 0o755); err != nil {
|
||||
return fmt.Errorf("%w: out dir: %v", ErrConversionFailed, err)
|
||||
}
|
||||
if err := copyFile(srcPath, filepath.Join(inDir, guestInputName)); err != nil {
|
||||
return fmt.Errorf("%w: stage input: %v", ErrConversionFailed, err)
|
||||
}
|
||||
|
||||
runCtx, cancel := context.WithTimeout(ctx, c.opts.Timeout)
|
||||
defer cancel()
|
||||
|
||||
stdout := &cappedWriter{max: c.opts.MaxLogBytes}
|
||||
stderr := &cappedWriter{max: c.opts.MaxLogBytes}
|
||||
fsConfig := wazero.NewFSConfig().
|
||||
WithFSMount(os.DirFS(inDir), guestInDir). // read-only, confined
|
||||
WithDirMount(outDir, guestOutDir) // writable
|
||||
modConfig := wazero.NewModuleConfig().
|
||||
WithArgs("mobitool", "-e", "-o", guestOutDir, guestInDir+"/"+guestInputName).
|
||||
WithFSConfig(fsConfig).
|
||||
WithStdout(stdout).
|
||||
WithStderr(stderr).
|
||||
WithSysWalltime().
|
||||
WithName("")
|
||||
|
||||
mod, instErr := c.runtime.InstantiateModule(runCtx, c.compiled, modConfig)
|
||||
if mod != nil {
|
||||
_ = mod.Close(runCtx)
|
||||
}
|
||||
combined := stdout.String() + "\n" + stderr.String()
|
||||
|
||||
if err := classifyRunError(ctx, runCtx, instErr, c.opts.Timeout, combined); err != nil {
|
||||
return err
|
||||
}
|
||||
// DRM and unconvertible layouts are reported on stdout with exit 0.
|
||||
if matchesAnyLine(combined, drmMarkers) {
|
||||
return ErrDRMProtected
|
||||
}
|
||||
if lineContains(combined, printReplicaMarker) {
|
||||
return fmt.Errorf("%w: print-replica (fixed-layout) book is not convertible", ErrConversionFailed)
|
||||
}
|
||||
|
||||
producedEpub := filepath.Join(outDir, guestEpubName)
|
||||
if fi, err := os.Stat(producedEpub); err != nil {
|
||||
return fmt.Errorf("%w: no EPUB produced: %s", ErrConversionFailed, trim(combined))
|
||||
} else if fi.Size() > c.opts.MaxOutputBytes {
|
||||
return fmt.Errorf("%w: output EPUB exceeds %d bytes", ErrConversionFailed, c.opts.MaxOutputBytes)
|
||||
}
|
||||
if err := validateEpub(producedEpub); err != nil {
|
||||
return fmt.Errorf("%w: %v", ErrConversionFailed, err)
|
||||
}
|
||||
if err := moveFile(producedEpub, dstPath); err != nil {
|
||||
return fmt.Errorf("%w: deliver output: %v", ErrConversionFailed, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// classifyRunError maps a wazero run error to our error taxonomy. Context
|
||||
// errors (timeout/cancel) MUST be checked before a generic nonzero exit code,
|
||||
// because WithCloseOnContextDone surfaces them as *sys.ExitError with special
|
||||
// codes that would otherwise read as an absurd "exit code".
|
||||
//
|
||||
// Ordering matters: the *caller's* context is checked first so a request-level
|
||||
// cancel OR deadline is propagated verbatim (never reclassified as a conversion
|
||||
// verdict — that would both mislead upstream cancellation and poison the
|
||||
// negative cache). Only once the caller's context is healthy do we attribute a
|
||||
// deadline to our own per-call timeout, which is a transient ErrConversionTimedOut.
|
||||
func classifyRunError(parent, run context.Context, instErr error, timeout time.Duration, combined string) error {
|
||||
if instErr == nil {
|
||||
return nil
|
||||
}
|
||||
// Caller's context ended (cancel or deadline) → propagate it verbatim.
|
||||
if parent.Err() != nil {
|
||||
return parent.Err()
|
||||
}
|
||||
// Our per-call timeout fired (caller's context is still healthy).
|
||||
if errors.Is(run.Err(), context.DeadlineExceeded) ||
|
||||
errors.Is(instErr, context.DeadlineExceeded) {
|
||||
return fmt.Errorf("%w: after %s", ErrConversionTimedOut, timeout)
|
||||
}
|
||||
if errors.Is(instErr, context.Canceled) {
|
||||
return context.Canceled
|
||||
}
|
||||
var exitErr *sys.ExitError
|
||||
if errors.As(instErr, &exitErr) {
|
||||
switch exitErr.ExitCode() {
|
||||
case sys.ExitCodeDeadlineExceeded:
|
||||
return fmt.Errorf("%w: after %s", ErrConversionTimedOut, timeout)
|
||||
case sys.ExitCodeContextCanceled:
|
||||
return context.Canceled
|
||||
case 0:
|
||||
return nil // normal exit; output checks follow
|
||||
default:
|
||||
return fmt.Errorf("%w: mobitool exit %d: %s", ErrConversionFailed, exitErr.ExitCode(), trim(combined))
|
||||
}
|
||||
}
|
||||
return fmt.Errorf("%w: run: %v", ErrConversionFailed, instErr)
|
||||
}
|
||||
|
||||
// validateEpub confirms a well-formed EPUB OCF container: a zip whose first
|
||||
// entry is a STORED (uncompressed) "mimetype" holding exactly
|
||||
// application/epub+zip, with a META-INF/container.xml that names an OPF
|
||||
// rootfile present in the archive.
|
||||
func validateEpub(path string) error {
|
||||
zr, err := zip.OpenReader(path)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open epub: %w", err)
|
||||
}
|
||||
defer zr.Close()
|
||||
|
||||
if len(zr.File) == 0 {
|
||||
return errors.New("empty epub")
|
||||
}
|
||||
first := zr.File[0]
|
||||
if first.Name != "mimetype" {
|
||||
return fmt.Errorf("first entry is %q, want mimetype", first.Name)
|
||||
}
|
||||
if first.Method != zip.Store {
|
||||
return errors.New("mimetype is not stored (uncompressed)")
|
||||
}
|
||||
rc, err := first.Open()
|
||||
if err != nil {
|
||||
return fmt.Errorf("read mimetype: %w", err)
|
||||
}
|
||||
mt, err := io.ReadAll(io.LimitReader(rc, 64))
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
return fmt.Errorf("read mimetype: %w", err)
|
||||
}
|
||||
if string(mt) != "application/epub+zip" {
|
||||
return fmt.Errorf("mimetype is %q", string(mt))
|
||||
}
|
||||
|
||||
names := make(map[string]bool, len(zr.File))
|
||||
for _, f := range zr.File {
|
||||
names[f.Name] = true
|
||||
}
|
||||
if !names["META-INF/container.xml"] {
|
||||
return errors.New("missing META-INF/container.xml")
|
||||
}
|
||||
rootfile, err := opfRootfile(zr)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !names[rootfile] {
|
||||
return fmt.Errorf("OPF rootfile %q missing from archive", rootfile)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func opfRootfile(zr *zip.ReadCloser) (string, error) {
|
||||
for _, f := range zr.File {
|
||||
if f.Name != "META-INF/container.xml" {
|
||||
continue
|
||||
}
|
||||
rc, err := f.Open()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("open container.xml: %w", err)
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(rc, 64<<10))
|
||||
rc.Close()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read container.xml: %w", err)
|
||||
}
|
||||
var c struct {
|
||||
Rootfiles []struct {
|
||||
FullPath string `xml:"full-path,attr"`
|
||||
} `xml:"rootfiles>rootfile"`
|
||||
}
|
||||
if err := xml.Unmarshal(data, &c); err != nil {
|
||||
return "", fmt.Errorf("parse container.xml: %w", err)
|
||||
}
|
||||
if len(c.Rootfiles) == 0 || c.Rootfiles[0].FullPath == "" {
|
||||
return "", errors.New("container.xml has no rootfile")
|
||||
}
|
||||
return c.Rootfiles[0].FullPath, nil
|
||||
}
|
||||
return "", errors.New("missing META-INF/container.xml")
|
||||
}
|
||||
|
||||
// matchesAnyLine reports whether any line of s contains any of the markers.
|
||||
func matchesAnyLine(s string, markers []string) bool {
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
for _, m := range markers {
|
||||
if strings.Contains(line, m) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func lineContains(s, marker string) bool {
|
||||
for _, line := range strings.Split(s, "\n") {
|
||||
if strings.Contains(line, marker) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// cappedWriter buffers up to max bytes and silently drops the rest, always
|
||||
// reporting a full write so the guest is never blocked on a full pipe.
|
||||
type cappedWriter struct {
|
||||
buf bytes.Buffer
|
||||
max int
|
||||
truncated bool
|
||||
}
|
||||
|
||||
func (w *cappedWriter) Write(p []byte) (int, error) {
|
||||
if room := w.max - w.buf.Len(); room > 0 {
|
||||
if room >= len(p) {
|
||||
w.buf.Write(p)
|
||||
} else {
|
||||
w.buf.Write(p[:room])
|
||||
w.truncated = true
|
||||
}
|
||||
} else if len(p) > 0 {
|
||||
w.truncated = true
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (w *cappedWriter) String() string { return w.buf.String() }
|
||||
|
||||
func copyFile(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer in.Close()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
out.Close()
|
||||
return err
|
||||
}
|
||||
return out.Close()
|
||||
}
|
||||
|
||||
// moveFile delivers src to dst atomically: rename when possible, else copy to a
|
||||
// temp file in dst's directory, fsync, and rename — never leaving a partial dst.
|
||||
func moveFile(src, dst string) error {
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := os.Rename(src, dst); err == nil {
|
||||
return nil
|
||||
}
|
||||
tmp, err := os.CreateTemp(filepath.Dir(dst), ".move-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
tmpName := tmp.Name()
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
tmp.Close()
|
||||
os.Remove(tmpName)
|
||||
return err
|
||||
}
|
||||
_, copyErr := io.Copy(tmp, in)
|
||||
in.Close()
|
||||
if copyErr == nil {
|
||||
copyErr = tmp.Sync()
|
||||
}
|
||||
if cerr := tmp.Close(); copyErr == nil {
|
||||
copyErr = cerr
|
||||
}
|
||||
if copyErr != nil {
|
||||
os.Remove(tmpName)
|
||||
return copyErr
|
||||
}
|
||||
if err := os.Rename(tmpName, dst); err != nil {
|
||||
os.Remove(tmpName)
|
||||
return err
|
||||
}
|
||||
return os.Remove(src)
|
||||
}
|
||||
|
||||
func trim(s string) string {
|
||||
const max = 500
|
||||
s = strings.TrimSpace(s)
|
||||
if len(s) > max {
|
||||
return s[:max] + "…"
|
||||
}
|
||||
return s
|
||||
}
|
||||
@@ -0,0 +1,247 @@
|
||||
package ebookconvert
|
||||
|
||||
import (
|
||||
"archive/zip"
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func newTestConverter(t *testing.T) *Converter {
|
||||
t.Helper()
|
||||
c, err := NewConverter(context.Background(), Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("NewConverter: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
||||
return c
|
||||
}
|
||||
|
||||
func TestConvert_DRMFreeProducesValidEpub(t *testing.T) {
|
||||
for _, fixture := range []string{"sample-ncx.mobi", "sample-cp1252.mobi"} {
|
||||
t.Run(fixture, func(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(context.Background(), filepath.Join("testdata", fixture), dst); err != nil {
|
||||
t.Fatalf("Convert: %v", err)
|
||||
}
|
||||
// Independently re-validate the delivered file is a real EPUB.
|
||||
assertValidEpub(t, dst)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_DRMProtectedRejected(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
err := c.Convert(context.Background(), filepath.Join("testdata", "sample-drm-v1.mobi"), dst)
|
||||
if !errors.Is(err, ErrDRMProtected) {
|
||||
t.Fatalf("got %v, want ErrDRMProtected", err)
|
||||
}
|
||||
if _, statErr := os.Stat(dst); !os.IsNotExist(statErr) {
|
||||
t.Fatalf("DRM source must not produce an output file, but %s exists", dst)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_OversizeRejected(t *testing.T) {
|
||||
c, err := NewConverter(context.Background(), Options{MaxSourceBytes: 1024})
|
||||
if err != nil {
|
||||
t.Fatalf("NewConverter: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrSourceTooLarge) {
|
||||
t.Fatalf("got %v, want ErrSourceTooLarge", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_CorruptSourceFails(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
bad := filepath.Join(t.TempDir(), "bad.mobi")
|
||||
if err := os.WriteFile(bad, []byte("not a mobi file at all"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(context.Background(), bad, dst); !errors.Is(err, ErrConversionFailed) {
|
||||
t.Fatalf("got %v, want ErrConversionFailed", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_MissingSourceFails(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(context.Background(), "testdata/does-not-exist.mobi", dst); !errors.Is(err, ErrConversionFailed) {
|
||||
t.Fatalf("got %v, want ErrConversionFailed", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_Concurrent(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
const n = 6
|
||||
errs := make(chan error, n)
|
||||
for i := 0; i < n; i++ {
|
||||
go func(i int) {
|
||||
dst := filepath.Join(t.TempDir(), "c.epub")
|
||||
errs <- c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
|
||||
}(i)
|
||||
}
|
||||
for i := 0; i < n; i++ {
|
||||
if err := <-errs; err != nil {
|
||||
t.Fatalf("concurrent Convert: %v", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_ContextCancelled(t *testing.T) {
|
||||
c := newTestConverter(t)
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel() // already canceled before the conversion starts
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(ctx, filepath.Join("testdata", "sample-ncx.mobi"), dst); err == nil {
|
||||
t.Fatal("expected error on canceled context")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateEpub_RejectsNonEpubZip(t *testing.T) {
|
||||
// A zip that is not an EPUB (no mimetype-first) must be rejected.
|
||||
p := filepath.Join(t.TempDir(), "plain.zip")
|
||||
f, err := os.Create(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
zw := zip.NewWriter(f)
|
||||
w, _ := zw.Create("hello.txt")
|
||||
_, _ = w.Write([]byte("hi"))
|
||||
_ = zw.Close()
|
||||
_ = f.Close()
|
||||
if err := validateEpub(p); err == nil {
|
||||
t.Fatal("expected validateEpub to reject a non-EPUB zip")
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateEpub_RejectsDeflatedMimetype(t *testing.T) {
|
||||
// mimetype present + correct content but DEFLATED (not stored) -> reject.
|
||||
p := filepath.Join(t.TempDir(), "bad.epub")
|
||||
f, err := os.Create(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
zw := zip.NewWriter(f)
|
||||
w, _ := zw.CreateHeader(&zip.FileHeader{Name: "mimetype", Method: zip.Deflate})
|
||||
_, _ = w.Write([]byte("application/epub+zip"))
|
||||
_ = zw.Close()
|
||||
_ = f.Close()
|
||||
if err := validateEpub(p); err == nil {
|
||||
t.Fatal("expected rejection of deflated mimetype")
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_AfterCloseReturnsUnavailable(t *testing.T) {
|
||||
c, err := NewConverter(context.Background(), Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("NewConverter: %v", err)
|
||||
}
|
||||
_ = c.Close(context.Background())
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
if err := c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst); !errors.Is(err, ErrUnavailable) {
|
||||
t.Fatalf("got %v, want ErrUnavailable", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConvert_TimeoutClassifiedNotRawExit(t *testing.T) {
|
||||
// A 1ns timeout must surface as ErrConversionTimedOut (a transient verdict,
|
||||
// distinct from a deterministic failure), never as a bogus "mobitool exit
|
||||
// <huge>" code.
|
||||
c, err := NewConverter(context.Background(), Options{Timeout: time.Nanosecond})
|
||||
if err != nil {
|
||||
t.Fatalf("NewConverter: %v", err)
|
||||
}
|
||||
t.Cleanup(func() { _ = c.Close(context.Background()) })
|
||||
dst := filepath.Join(t.TempDir(), "out.epub")
|
||||
err = c.Convert(context.Background(), filepath.Join("testdata", "sample-ncx.mobi"), dst)
|
||||
if !errors.Is(err, ErrConversionTimedOut) {
|
||||
t.Fatalf("got %v, want ErrConversionTimedOut", err)
|
||||
}
|
||||
if errors.Is(err, ErrConversionFailed) {
|
||||
t.Fatalf("a timeout must not also satisfy ErrConversionFailed (it would be negatively cached): %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "exit ") {
|
||||
t.Fatalf("timeout leaked a raw exit code: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// classifyRunError must propagate a caller's deadline as context.DeadlineExceeded
|
||||
// (CR review): reclassifying it as a conversion failure breaks upstream
|
||||
// cancellation handling and would poison the negative cache.
|
||||
func TestClassifyRunError_ParentDeadlinePropagates(t *testing.T) {
|
||||
parent, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
|
||||
defer cancel()
|
||||
run := parent // run derives from parent, so it is also past-deadline
|
||||
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
|
||||
if !errors.Is(err, context.DeadlineExceeded) {
|
||||
t.Fatalf("got %v, want context.DeadlineExceeded propagated", err)
|
||||
}
|
||||
if errors.Is(err, ErrConversionFailed) || errors.Is(err, ErrConversionTimedOut) {
|
||||
t.Fatalf("a caller deadline must not be reclassified as a conversion verdict: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// When the caller's context is healthy but our own per-call timeout fired, the
|
||||
// result is the transient ErrConversionTimedOut sentinel (not ErrConversionFailed).
|
||||
func TestClassifyRunError_OwnTimeoutIsTimedOutSentinel(t *testing.T) {
|
||||
parent := context.Background() // caller healthy
|
||||
run, cancel := context.WithDeadline(context.Background(), time.Now().Add(-time.Hour))
|
||||
defer cancel()
|
||||
err := classifyRunError(parent, run, context.DeadlineExceeded, time.Minute, "mobitool output")
|
||||
if !errors.Is(err, ErrConversionTimedOut) {
|
||||
t.Fatalf("got %v, want ErrConversionTimedOut", err)
|
||||
}
|
||||
if errors.Is(err, ErrConversionFailed) {
|
||||
t.Fatalf("own timeout must not satisfy ErrConversionFailed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func assertValidEpub(t *testing.T, path string) {
|
||||
t.Helper()
|
||||
zr, err := zip.OpenReader(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open epub: %v", err)
|
||||
}
|
||||
defer zr.Close()
|
||||
if len(zr.File) == 0 || zr.File[0].Name != "mimetype" {
|
||||
t.Fatalf("first entry not mimetype")
|
||||
}
|
||||
rc, _ := zr.File[0].Open()
|
||||
mt, _ := io.ReadAll(rc)
|
||||
rc.Close()
|
||||
if strings.TrimSpace(string(mt)) != "application/epub+zip" {
|
||||
t.Fatalf("bad mimetype %q", mt)
|
||||
}
|
||||
var hasContainer, hasContent bool
|
||||
for _, fl := range zr.File {
|
||||
if fl.Name == "META-INF/container.xml" {
|
||||
hasContainer = true
|
||||
}
|
||||
if strings.HasPrefix(fl.Name, "OEBPS/") {
|
||||
hasContent = true
|
||||
}
|
||||
}
|
||||
if !hasContainer {
|
||||
t.Fatal("missing META-INF/container.xml")
|
||||
}
|
||||
if !hasContent {
|
||||
t.Fatal("no OEBPS content")
|
||||
}
|
||||
}
|
||||
|
||||
// Guard so the default timeout constant stays sane if edited.
|
||||
func TestDefaults(t *testing.T) {
|
||||
if DefaultTimeout < time.Second {
|
||||
t.Fatal("DefaultTimeout too small")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
// Package ebookconvert converts Kindle-family ebooks (MOBI/AZW/AZW3) to EPUB
|
||||
// in-process, by running libmobi's mobitool compiled to wasm32-wasi on the
|
||||
// wazero runtime. No cgo, no external binary; untrusted input is parsed inside
|
||||
// the WASM sandbox. See docs/.../2026-06-17-kindle-epub-conversion-design.md.
|
||||
package ebookconvert
|
||||
|
||||
import "errors"
|
||||
|
||||
var (
|
||||
// ErrDRMProtected means the source is encrypted/DRM'd and cannot be
|
||||
// converted without a device key. Callers should surface the
|
||||
// "open externally" path; the source is never usable in-app.
|
||||
ErrDRMProtected = errors.New("ebookconvert: source is DRM-protected")
|
||||
|
||||
// ErrConversionFailed means mobitool failed or produced output that is
|
||||
// not a valid EPUB (corrupt source, unexpected format, internal error).
|
||||
// It is a *deterministic* verdict for a given input — the same bytes will
|
||||
// fail the same way — so the cache may remember it negatively.
|
||||
ErrConversionFailed = errors.New("ebookconvert: conversion failed")
|
||||
|
||||
// ErrConversionTimedOut means the converter's own per-call timeout fired.
|
||||
// Unlike ErrConversionFailed it is *transient* (it reflects load/size, not
|
||||
// the input being unconvertible), so it is NOT negatively cached and the
|
||||
// next read may retry. It deliberately does not wrap ErrConversionFailed.
|
||||
ErrConversionTimedOut = errors.New("ebookconvert: conversion timed out")
|
||||
|
||||
// ErrSourceTooLarge means the source exceeds the configured size cap and
|
||||
// was rejected before invoking the converter.
|
||||
ErrSourceTooLarge = errors.New("ebookconvert: source exceeds max size")
|
||||
|
||||
// ErrUnavailable means the converter could not be initialized (the
|
||||
// embedded module failed to compile/instantiate). The capability must
|
||||
// not be advertised in this state.
|
||||
ErrUnavailable = errors.New("ebookconvert: converter unavailable")
|
||||
)
|
||||
Executable
BIN
Binary file not shown.
@@ -0,0 +1 @@
|
||||
38a544c5f516a6b51c9eba21f7f06e964de55aabdd05c3bbe7825f08e6f88303 /out/mobitool.wasm
|
||||
@@ -0,0 +1,33 @@
|
||||
package ebookconvert
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
// The committed mobitool.wasm must match the sha256 recorded next to it. This
|
||||
// is the cheap half of the artifact guardrail: together with the real fixture
|
||||
// conversions in converter_test.go (the "smoke test"), it ensures a careless
|
||||
// artifact swap or a corrupted checkout is caught in CI instead of shipping a
|
||||
// mystery binary. The sidecar is sha256sum format (`<hex> <path>`); we compare
|
||||
// only the hash, since the recorded path is build-environment specific.
|
||||
func TestEmbeddedWasmMatchesRecordedSHA256(t *testing.T) {
|
||||
raw, err := os.ReadFile("mobitool.wasm.sha256")
|
||||
if err != nil {
|
||||
t.Fatalf("read sha256 sidecar: %v", err)
|
||||
}
|
||||
fields := strings.Fields(string(raw))
|
||||
if len(fields) == 0 {
|
||||
t.Fatal("mobitool.wasm.sha256 is empty")
|
||||
}
|
||||
want := strings.ToLower(fields[0])
|
||||
|
||||
sum := sha256.Sum256(mobitoolWasm)
|
||||
got := hex.EncodeToString(sum[:])
|
||||
if got != want {
|
||||
t.Fatalf("embedded mobitool.wasm sha256 = %s, recorded = %s — artifact and provenance disagree", got, want)
|
||||
}
|
||||
}
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Test fixtures
|
||||
|
||||
These are libmobi's own test sample files, vendored for converter unit tests.
|
||||
|
||||
Source: https://github.com/bfabiszewski/libmobi `tests/samples/` (LGPL-3.0; the
|
||||
samples are project test data authored by the libmobi project).
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `sample-ncx.mobi` | DRM-free KF8/MOBI6 hybrid — happy-path conversion |
|
||||
| `sample-cp1252.mobi` | DRM-free cp1252-encoded — second happy-path encoding |
|
||||
| `sample-drm-v1.mobi` | Encrypted source — exercises the DRM rejection path |
|
||||
Binary file not shown.
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,84 @@
|
||||
# Builds `mobitool.wasm` — libmobi's mobitool compiled to wasm32-wasi — used by
|
||||
# the in-process Kindle->EPUB converter (internal/ebookconvert) via wazero.
|
||||
#
|
||||
# The output is architecture-independent wasm32 bytecode: build it on any host
|
||||
# (CI is amd64), run it on any server arch through wazero (pure Go).
|
||||
#
|
||||
# Build + extract the artifact:
|
||||
# docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm
|
||||
# id=$(docker create mobitool-wasm); docker cp "$id:/out/mobitool.wasm" \
|
||||
# internal/ebookconvert/mobitool.wasm; docker rm "$id"
|
||||
#
|
||||
# Pins (bump deliberately; the converter cache key includes these):
|
||||
# wasi-sdk 25.0 · libmobi 906274205c11944b628da1c553b255acb1af7c55 · zlib 1.3.1
|
||||
#
|
||||
# Proven build config (see ../../docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md):
|
||||
# --with-libxml2=no -> internal xmlwriter provides OPF/EPUB support (-e)
|
||||
# zlib built to wasm -> avoids libmobi's bundled-miniz duplicate-symbol clash
|
||||
# with mobitool's own miniz (EPUB zip) at wasm-ld time.
|
||||
|
||||
FROM debian:bookworm-slim AS build
|
||||
|
||||
ARG WASI_SDK_VERSION=25
|
||||
ARG LIBMOBI_COMMIT=906274205c11944b628da1c553b255acb1af7c55
|
||||
ARG ZLIB_VERSION=1.3.1
|
||||
|
||||
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
|
||||
build-essential autoconf automake libtool pkg-config git curl ca-certificates xz-utils \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# --- wasi-sdk (clang + wasm32-wasi sysroot) ---
|
||||
RUN cd /opt \
|
||||
&& curl -fsSL -o wasi-sdk.tar.gz \
|
||||
"https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux.tar.gz" \
|
||||
&& tar xzf wasi-sdk.tar.gz && mv "wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux" wasi-sdk && rm wasi-sdk.tar.gz
|
||||
ENV WASI=/opt/wasi-sdk
|
||||
ENV SYSROOT=/opt/wasi-sdk/share/wasi-sysroot
|
||||
|
||||
# --- zlib -> wasm (compile sources directly; zlib's configure runs target binaries) ---
|
||||
RUN cd /opt \
|
||||
&& curl -fsSL -o zlib.tar.gz "https://github.com/madler/zlib/releases/download/v${ZLIB_VERSION}/zlib-${ZLIB_VERSION}.tar.gz" \
|
||||
&& tar xzf zlib.tar.gz && cd "zlib-${ZLIB_VERSION}" \
|
||||
&& for s in adler32 compress crc32 deflate gzclose gzlib gzread gzwrite infback inffast inflate inftrees trees uncompr zutil; do \
|
||||
"$WASI/bin/clang" -O2 --sysroot="$SYSROOT" -DHAVE_UNISTD_H -c "$s.c" -o "$s.o"; done \
|
||||
&& "$WASI/bin/llvm-ar" rcs libz.a *.o \
|
||||
&& mkdir -p /opt/zlibwasm/lib /opt/zlibwasm/include \
|
||||
&& cp libz.a /opt/zlibwasm/lib/ && cp zlib.h zconf.h /opt/zlibwasm/include/
|
||||
|
||||
# --- libmobi + mobitool -> wasm ---
|
||||
RUN mkdir -p /work && cd /work \
|
||||
&& git clone https://github.com/bfabiszewski/libmobi.git && cd libmobi \
|
||||
&& git checkout "${LIBMOBI_COMMIT}" \
|
||||
&& ./autogen.sh \
|
||||
&& ./configure --host=wasm32-wasi \
|
||||
--with-libxml2=no --disable-shared --enable-static \
|
||||
CC="$WASI/bin/clang" AR="$WASI/bin/llvm-ar" RANLIB="$WASI/bin/llvm-ranlib" \
|
||||
CFLAGS="-O2 --sysroot=$SYSROOT -I/opt/zlibwasm/include" \
|
||||
CPPFLAGS="-I/opt/zlibwasm/include" \
|
||||
LDFLAGS="--sysroot=$SYSROOT -L/opt/zlibwasm/lib" \
|
||||
LIBS="-lz" \
|
||||
&& make -j"$(nproc)" \
|
||||
&& mkdir -p /out && cp tools/mobitool /out/mobitool.wasm \
|
||||
&& sha256sum /out/mobitool.wasm | tee /out/mobitool.wasm.sha256
|
||||
|
||||
# Smoke test: convert a bundled libmobi sample to EPUB using a pinned,
|
||||
# checksum-verified wasmtime — NOT the `curl | bash` convenience installer, which
|
||||
# is documented as interactive-only and unsafe/non-deterministic for builds — so
|
||||
# the build fails loudly if the wasm can't actually convert.
|
||||
ARG WASMTIME_VERSION=27.0.0
|
||||
ARG WASMTIME_SHA256=74678f6ec49d8b858ca5181d5492b967245f6c072b5763ea14a75eee1ad3c2d7
|
||||
RUN cd /opt \
|
||||
&& curl -fsSL -o wasmtime.tar.xz \
|
||||
"https://github.com/bytecodealliance/wasmtime/releases/download/v${WASMTIME_VERSION}/wasmtime-v${WASMTIME_VERSION}-x86_64-linux.tar.xz" \
|
||||
&& echo "${WASMTIME_SHA256} wasmtime.tar.xz" | sha256sum -c - \
|
||||
&& tar xJf wasmtime.tar.xz && rm wasmtime.tar.xz \
|
||||
&& WASMTIME="/opt/wasmtime-v${WASMTIME_VERSION}-x86_64-linux/wasmtime" \
|
||||
&& mkdir -p /smoke/out \
|
||||
&& cp /work/libmobi/tests/samples/sample-ncx.mobi /smoke/in.mobi \
|
||||
&& "$WASMTIME" run --dir=/smoke::/smoke /out/mobitool.wasm -e -o /smoke/out /smoke/in.mobi \
|
||||
&& test -f /smoke/out/in.epub \
|
||||
&& echo "SMOKE OK: $(wc -c < /smoke/out/in.epub) byte EPUB produced"
|
||||
|
||||
FROM scratch AS artifact
|
||||
COPY --from=build /out/mobitool.wasm /mobitool.wasm
|
||||
COPY --from=build /out/mobitool.wasm.sha256 /mobitool.wasm.sha256
|
||||
@@ -0,0 +1,64 @@
|
||||
# mobitool.wasm — Kindle→EPUB converter module
|
||||
|
||||
`mobitool.wasm` is [libmobi](https://github.com/bfabiszewski/libmobi)'s `mobitool`
|
||||
compiled to `wasm32-wasi`. The server runs it **in-process** via
|
||||
[`wazero`](https://github.com/tetratelabs/wazero) (pure Go) to convert
|
||||
MOBI/AZW/AZW3 ebooks to EPUB on demand. See the design doc:
|
||||
`docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md`.
|
||||
|
||||
## Why this shape
|
||||
- **No cgo, no external binary.** The `.wasm` is `go:embed`-ed; the Go build stays
|
||||
`CGO_ENABLED=0` and cross-compiles trivially.
|
||||
- **Architecture-independent.** wasm32 bytecode runs identically on amd64/arm64
|
||||
servers through wazero. Build once (CI, amd64), run anywhere.
|
||||
- **Sandboxed.** Untrusted ebook bytes are parsed inside the WASM sandbox with
|
||||
only a per-conversion scratch dir mounted — a libmobi memory bug on a malicious
|
||||
file cannot reach the host.
|
||||
|
||||
## Pinned versions
|
||||
|
||||
| Component | Version / commit |
|
||||
|-------------------|--------------------------------------------|
|
||||
| wasi-sdk | 25.0 |
|
||||
| libmobi | `906274205c11944b628da1c553b255acb1af7c55` |
|
||||
| zlib | 1.3.1 (compiled to wasm) |
|
||||
| wasmtime (smoke) | 27.0.0 (build-time smoke test only) |
|
||||
|
||||
License: libmobi is **LGPL-3.0-or-later**; shipping the unmodified compiled
|
||||
artifact + this build recipe satisfies the relink obligation. zlib is zlib-license.
|
||||
|
||||
## Build config notes (validated by spike, 2026-06-17 on linux/amd64)
|
||||
- `--with-libxml2=no` → libmobi's **internal xmlwriter** provides OPF/EPUB output,
|
||||
so the `-e` (create EPUB) path works with no libxml2 dependency.
|
||||
- libmobi is built **against a wasm-compiled zlib** (not `--with-zlib=no`). Forcing
|
||||
`--with-zlib=no` makes libmobi vendor its own miniz, which then collides with
|
||||
`mobitool`'s separate miniz (used for EPUB zip creation) — `wasm-ld` rejects the
|
||||
duplicate symbols. Real zlib for the library + the tool's miniz for zip = no clash.
|
||||
|
||||
## Rebuild + install the artifact
|
||||
```sh
|
||||
docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm
|
||||
id=$(docker create mobitool-wasm)
|
||||
docker cp "$id:/mobitool.wasm" internal/ebookconvert/mobitool.wasm
|
||||
docker cp "$id:/mobitool.wasm.sha256" internal/ebookconvert/mobitool.wasm.sha256
|
||||
docker rm "$id"
|
||||
```
|
||||
The build runs a **smoke conversion** of a bundled libmobi sample and fails if no
|
||||
EPUB is produced. `mobitool -v` embeds `__DATE__`/`__TIME__`, so rebuilds are not
|
||||
byte-for-byte reproducible — rely on the smoke test + recorded sha256, not a
|
||||
bit-identical rebuild check.
|
||||
|
||||
## Runtime behavior characterized by the spike
|
||||
- Converts MOBI6, KF8/AZW3 hybrids, HUFF/CDIC-compressed, and unicode samples to
|
||||
well-formed EPUB (`mimetype` first = `application/epub+zip`, `META-INF/container.xml`,
|
||||
`OEBPS/*`).
|
||||
- **DRM:** `mobitool` prints `Document is encrypted` to **stdout** but still **exits 0**
|
||||
and writes an EPUB (garbage content when it lacks the key). So the converter must
|
||||
capture stdout and validate the output is a usable EPUB — exit code alone is not a
|
||||
reliable DRM/failure signal.
|
||||
- ⚠️ **Brittleness note:** DRM/print-replica detection matches the **English
|
||||
stdout strings** mobitool emits (`drmMarkers` / `printReplicaMarker` in
|
||||
`internal/ebookconvert/converter.go`). When bumping the pinned libmobi commit,
|
||||
re-check those messages still match — a wording change silently downgrades a
|
||||
DRM book to a generic "no EPUB produced" failure (it still falls back to the raw
|
||||
original safely, just without the explicit DRM signal).
|
||||
Reference in New Issue
Block a user