e99079abf8
* Kindle->EPUB conversion: design + proven wasm build pipeline
Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.
This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
9 review fixes folded in: failure contract, strong cache key + negative
cache, wazero command-module specifics, FS-sandbox tightening,
double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
built on amd64. go:embed target for the converter package (next).
Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.
Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: converter core + cache (Codex-reviewed)
internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.
18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).
Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
FS isolation ultimately relies on running as a non-root user (memory-safety
is the WASM boundary). validateEpub now requires STORED mimetype + verifies
the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint
Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.
- handlers.EbookConversion (converter + per-request flag predicate) on the read
handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
served_format, header contract}; enabled only when flag on AND converter
wired (double gate) so the Android client can decide whether to flip
mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
if it fails), cache dir is a sibling of TranscodeDir, flag read per request.
Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification
Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.
Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
the per-call timeout as transient and propagate a caller's cancel/deadline
verbatim instead of reclassifying it as a conversion failure. remember() now
only caches deterministic verdicts (DRM / failed), so a one-off timeout under
load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
context.WithoutCancel), so one caller cancelling no longer aborts the shared
work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
not FIFO.
Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
a negatively-cached source serves the failed contract, a miss advertises the
converted representation cheaply without triggering a (minute-long, ~1 GiB)
conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
capability endpoint no longer hit the DB per request.
Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
+ provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).
Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: remove ebookconvert workflow
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
85 lines
4.5 KiB
Docker
85 lines
4.5 KiB
Docker
# Builds `mobitool.wasm` — libmobi's mobitool compiled to wasm32-wasi — used by
|
|
# the in-process Kindle->EPUB converter (internal/ebookconvert) via wazero.
|
|
#
|
|
# The output is architecture-independent wasm32 bytecode: build it on any host
|
|
# (CI is amd64), run it on any server arch through wazero (pure Go).
|
|
#
|
|
# Build + extract the artifact:
|
|
# docker build --platform linux/amd64 -t mobitool-wasm tools/mobitool-wasm
|
|
# id=$(docker create mobitool-wasm); docker cp "$id:/out/mobitool.wasm" \
|
|
# internal/ebookconvert/mobitool.wasm; docker rm "$id"
|
|
#
|
|
# Pins (bump deliberately; the converter cache key includes these):
|
|
# wasi-sdk 25.0 · libmobi 906274205c11944b628da1c553b255acb1af7c55 · zlib 1.3.1
|
|
#
|
|
# Proven build config (see ../../docs/superpowers/specs/2026-06-17-kindle-epub-conversion-design.md):
|
|
# --with-libxml2=no -> internal xmlwriter provides OPF/EPUB support (-e)
|
|
# zlib built to wasm -> avoids libmobi's bundled-miniz duplicate-symbol clash
|
|
# with mobitool's own miniz (EPUB zip) at wasm-ld time.
|
|
|
|
FROM debian:bookworm-slim AS build
|
|
|
|
ARG WASI_SDK_VERSION=25
|
|
ARG LIBMOBI_COMMIT=906274205c11944b628da1c553b255acb1af7c55
|
|
ARG ZLIB_VERSION=1.3.1
|
|
|
|
RUN apt-get update -qq && DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --no-install-recommends \
|
|
build-essential autoconf automake libtool pkg-config git curl ca-certificates xz-utils \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# --- wasi-sdk (clang + wasm32-wasi sysroot) ---
|
|
RUN cd /opt \
|
|
&& curl -fsSL -o wasi-sdk.tar.gz \
|
|
"https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-${WASI_SDK_VERSION}/wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux.tar.gz" \
|
|
&& tar xzf wasi-sdk.tar.gz && mv "wasi-sdk-${WASI_SDK_VERSION}.0-x86_64-linux" wasi-sdk && rm wasi-sdk.tar.gz
|
|
ENV WASI=/opt/wasi-sdk
|
|
ENV SYSROOT=/opt/wasi-sdk/share/wasi-sysroot
|
|
|
|
# --- zlib -> wasm (compile sources directly; zlib's configure runs target binaries) ---
|
|
RUN cd /opt \
|
|
&& curl -fsSL -o zlib.tar.gz "https://github.com/madler/zlib/releases/download/v${ZLIB_VERSION}/zlib-${ZLIB_VERSION}.tar.gz" \
|
|
&& tar xzf zlib.tar.gz && cd "zlib-${ZLIB_VERSION}" \
|
|
&& for s in adler32 compress crc32 deflate gzclose gzlib gzread gzwrite infback inffast inflate inftrees trees uncompr zutil; do \
|
|
"$WASI/bin/clang" -O2 --sysroot="$SYSROOT" -DHAVE_UNISTD_H -c "$s.c" -o "$s.o"; done \
|
|
&& "$WASI/bin/llvm-ar" rcs libz.a *.o \
|
|
&& mkdir -p /opt/zlibwasm/lib /opt/zlibwasm/include \
|
|
&& cp libz.a /opt/zlibwasm/lib/ && cp zlib.h zconf.h /opt/zlibwasm/include/
|
|
|
|
# --- libmobi + mobitool -> wasm ---
|
|
RUN mkdir -p /work && cd /work \
|
|
&& git clone https://github.com/bfabiszewski/libmobi.git && cd libmobi \
|
|
&& git checkout "${LIBMOBI_COMMIT}" \
|
|
&& ./autogen.sh \
|
|
&& ./configure --host=wasm32-wasi \
|
|
--with-libxml2=no --disable-shared --enable-static \
|
|
CC="$WASI/bin/clang" AR="$WASI/bin/llvm-ar" RANLIB="$WASI/bin/llvm-ranlib" \
|
|
CFLAGS="-O2 --sysroot=$SYSROOT -I/opt/zlibwasm/include" \
|
|
CPPFLAGS="-I/opt/zlibwasm/include" \
|
|
LDFLAGS="--sysroot=$SYSROOT -L/opt/zlibwasm/lib" \
|
|
LIBS="-lz" \
|
|
&& make -j"$(nproc)" \
|
|
&& mkdir -p /out && cp tools/mobitool /out/mobitool.wasm \
|
|
&& sha256sum /out/mobitool.wasm | tee /out/mobitool.wasm.sha256
|
|
|
|
# Smoke test: convert a bundled libmobi sample to EPUB using a pinned,
|
|
# checksum-verified wasmtime — NOT the `curl | bash` convenience installer, which
|
|
# is documented as interactive-only and unsafe/non-deterministic for builds — so
|
|
# the build fails loudly if the wasm can't actually convert.
|
|
ARG WASMTIME_VERSION=27.0.0
|
|
ARG WASMTIME_SHA256=74678f6ec49d8b858ca5181d5492b967245f6c072b5763ea14a75eee1ad3c2d7
|
|
RUN cd /opt \
|
|
&& curl -fsSL -o wasmtime.tar.xz \
|
|
"https://github.com/bytecodealliance/wasmtime/releases/download/v${WASMTIME_VERSION}/wasmtime-v${WASMTIME_VERSION}-x86_64-linux.tar.xz" \
|
|
&& echo "${WASMTIME_SHA256} wasmtime.tar.xz" | sha256sum -c - \
|
|
&& tar xJf wasmtime.tar.xz && rm wasmtime.tar.xz \
|
|
&& WASMTIME="/opt/wasmtime-v${WASMTIME_VERSION}-x86_64-linux/wasmtime" \
|
|
&& mkdir -p /smoke/out \
|
|
&& cp /work/libmobi/tests/samples/sample-ncx.mobi /smoke/in.mobi \
|
|
&& "$WASMTIME" run --dir=/smoke::/smoke /out/mobitool.wasm -e -o /smoke/out /smoke/in.mobi \
|
|
&& test -f /smoke/out/in.epub \
|
|
&& echo "SMOKE OK: $(wc -c < /smoke/out/in.epub) byte EPUB produced"
|
|
|
|
FROM scratch AS artifact
|
|
COPY --from=build /out/mobitool.wasm /mobitool.wasm
|
|
COPY --from=build /out/mobitool.wasm.sha256 /mobitool.wasm.sha256
|