* Kindle->EPUB conversion: design + proven wasm build pipeline
Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.
This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
9 review fixes folded in: failure contract, strong cache key + negative
cache, wazero command-module specifics, FS-sandbox tightening,
double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
built on amd64. go:embed target for the converter package (next).
Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.
Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: converter core + cache (Codex-reviewed)
internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.
18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).
Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
FS isolation ultimately relies on running as a non-root user (memory-safety
is the WASM boundary). validateEpub now requires STORED mimetype + verifies
the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint
Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.
- handlers.EbookConversion (converter + per-request flag predicate) on the read
handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
served_format, header contract}; enabled only when flag on AND converter
wired (double gate) so the Android client can decide whether to flip
mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
if it fails), cache dir is a sibling of TranscodeDir, flag read per request.
Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification
Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.
Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
the per-call timeout as transient and propagate a caller's cancel/deadline
verbatim instead of reclassifying it as a conversion failure. remember() now
only caches deterministic verdicts (DRM / failed), so a one-off timeout under
load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
context.WithoutCancel), so one caller cancelling no longer aborts the shared
work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
not FIFO.
Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
a negatively-cached source serves the failed contract, a miss advertises the
converted representation cheaply without triggering a (minute-long, ~1 GiB)
conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
capability endpoint no longer hit the DB per request.
Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
+ provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).
Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* ci: remove ebookconvert workflow
---------
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
104 lines
3.8 KiB
Go
104 lines
3.8 KiB
Go
package api
|
||
|
||
import (
|
||
"context"
|
||
"log/slog"
|
||
"os"
|
||
"path/filepath"
|
||
"strings"
|
||
"sync"
|
||
"time"
|
||
|
||
"github.com/Silo-Server/silo-server/internal/api/handlers"
|
||
"github.com/Silo-Server/silo-server/internal/catalog"
|
||
"github.com/Silo-Server/silo-server/internal/config"
|
||
"github.com/Silo-Server/silo-server/internal/ebookconvert"
|
||
)
|
||
|
||
// ebookKindleConversionSettingKey is the admin flag that gates Kindle->EPUB
|
||
// conversion. Off by default; toggled via PUT /admin/settings/{key}.
|
||
const ebookKindleConversionSettingKey = "ebook.kindle_conversion_enabled"
|
||
|
||
// buildEbookConversion wires the in-process Kindle->EPUB converter for the read
|
||
// handler. It compiles the embedded WASM module once at startup; if that fails
|
||
// the feature stays off (returns nil) and the raw original is served as before.
|
||
// The admin flag is read with a short TTL (see ebookFlagPredicate), so it can be
|
||
// toggled without a restart yet does not hit the DB on every read.
|
||
//
|
||
// The Converter (a wazero runtime, ~no external resources — per-conversion
|
||
// scratch dirs are cleaned as they go) is intentionally owned for the process
|
||
// lifetime: it is not Close()d on shutdown because process exit reclaims it and
|
||
// there is nothing to flush. Memory is bounded per conversion by
|
||
// ebookconvert.DefaultMaxMemoryPages × DefaultConcurrency (see the design doc's
|
||
// resource notes); tune via ebookconvert.Options if the deployment is tight.
|
||
func buildEbookConversion(deps Dependencies, settings catalog.SettingsStore) *handlers.EbookConversion {
|
||
if settings == nil {
|
||
return nil
|
||
}
|
||
converter, err := ebookconvert.NewConverter(context.Background(), ebookconvert.Options{})
|
||
if err != nil {
|
||
slog.Warn("ebook Kindle->EPUB conversion unavailable: converter init failed", "error", err)
|
||
return nil
|
||
}
|
||
cache, err := ebookconvert.NewCache(converter, ebookconvert.CacheOptions{Dir: ebookConversionCacheDir(deps.CurrentConfig())})
|
||
if err != nil {
|
||
slog.Warn("ebook Kindle->EPUB conversion unavailable: cache init failed", "error", err)
|
||
_ = converter.Close(context.Background())
|
||
return nil
|
||
}
|
||
slog.Info("ebook Kindle->EPUB conversion ready (admin-flag gated)", "setting", ebookKindleConversionSettingKey)
|
||
return &handlers.EbookConversion{
|
||
Converter: cache,
|
||
Enabled: ebookFlagPredicate(settings, ebookKindleConversionSettingKey),
|
||
}
|
||
}
|
||
|
||
// ebookFlagPredicate returns a cached boolean reader for a server setting. The
|
||
// read path (every ebook read + the capability endpoint, which clients may
|
||
// poll) would otherwise issue a DB query per request; a short TTL collapses
|
||
// bursts to one query while keeping admin toggles effectively instant. On a read
|
||
// error it keeps the last known value rather than flapping the feature off.
|
||
func ebookFlagPredicate(settings catalog.SettingsStore, key string) func(context.Context) bool {
|
||
const ttl = 3 * time.Second
|
||
var (
|
||
mu sync.Mutex
|
||
value bool
|
||
fetched time.Time
|
||
hasVal bool
|
||
)
|
||
return func(ctx context.Context) bool {
|
||
mu.Lock()
|
||
defer mu.Unlock()
|
||
if hasVal && time.Since(fetched) < ttl {
|
||
return value
|
||
}
|
||
v, err := settings.Get(ctx, key)
|
||
if err != nil {
|
||
return value // keep last known (false until the first successful read)
|
||
}
|
||
value = isTruthySetting(v)
|
||
fetched = time.Now()
|
||
hasVal = true
|
||
return value
|
||
}
|
||
}
|
||
|
||
// ebookConversionCacheDir derives the converted-EPUB cache directory, a sibling
|
||
// of the transcode dir so it lives alongside other derived media.
|
||
func ebookConversionCacheDir(cfg *config.Config) string {
|
||
base := os.TempDir()
|
||
if cfg != nil && strings.TrimSpace(cfg.Playback.TranscodeDir) != "" {
|
||
base = filepath.Dir(cfg.Playback.TranscodeDir)
|
||
}
|
||
return filepath.Join(base, "silo-ebook-epub")
|
||
}
|
||
|
||
func isTruthySetting(v string) bool {
|
||
switch strings.ToLower(strings.TrimSpace(v)) {
|
||
case "true", "1", "yes", "on", "enabled":
|
||
return true
|
||
default:
|
||
return false
|
||
}
|
||
}
|