haeaeae/silo-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ed6c084c68080f09bb2243ec4fcea4e1fb83dca5
silo-server/go.mod
T
RXWatcherGitHubClaude CodeQuick
 e99079abf8 Server-side Kindle→EPUB conversion (mobi/azw/azw3) for in-app reading (#171) 
* Kindle->EPUB conversion: design + proven wasm build pipeline

Server-side MOBI/AZW/AZW3 -> EPUB conversion so the Android in-app reader
can render Kindle-family ebooks. Conversion runs in-process via libmobi's
mobitool compiled to wasm32-wasi, executed by wazero (pure Go) -- no cgo,
no external binary, arch-independent, sandboxed untrusted input.

This commit lands the design + the validated build artifact (spike done):
- docs/.../2026-06-17-kindle-epub-conversion-design.md (Codex-reviewed;
  9 review fixes folded in: failure contract, strong cache key + negative
  cache, wazero command-module specifics, FS-sandbox tightening,
  double-gated capability, serve headers, .wasm guardrails).
- tools/mobitool-wasm/{Dockerfile,README.md}: reproducible build of
  mobitool.wasm (wasi-sdk 25, libmobi 9062742, zlib 1.3.1->wasm), with a
  smoke-conversion gate. Build proven on native amd64.
- internal/ebookconvert/mobitool.wasm (+ .sha256): canonical artifact,
  built on amd64. go:embed target for the converter package (next).

Spike proven on amd64: -e EPUB path works with --with-libxml2=no (internal
xmlwriter); converts MOBI6/KF8/HUFF-CDIC/unicode -> well-formed EPUB;
verified end-to-end under wazero (WASI preopen + argv + _start). Build
gotcha: link libmobi against real (wasm) zlib, not --with-zlib=no, to avoid
miniz duplicate-symbol clash with mobitool's zip miniz. DRM gotcha:
mobitool prints "Document is encrypted" to stdout but exits 0 -> detect via
stdout + output validation, not exit code.

Not yet implemented: internal/ebookconvert Go package (wazero harness +
cache + singleflight), read-handler wiring, admin flag, client capability.
v1-scope proposal required before PR.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: converter core + cache (Codex-reviewed)

internal/ebookconvert: in-process MOBI/AZW/AZW3 -> EPUB via the embedded
mobitool.wasm on wazero. Converter compiles the module once and instantiates
per conversion (isolated). Cache adds on-disk, singleflighted, size-bounded,
negative-cached conversion keyed by file identity + module fingerprint.

18 tests pass (DRM-free->valid EPUB, DRM->ErrDRMProtected + no output,
oversize/corrupt/missing/timeout/cancel/after-close, 6/8-way concurrent,
EPUB structural validation incl. stored-mimetype + container rootfile,
cache miss/hit/key-change/singleflight/eviction/negative-cache).

Codex review fixes folded in:
- timeout/cancel classified before generic nonzero exit (WithCloseOnContextDone
  surfaces sys.ExitError special codes); no more bogus "exit <huge>".
- DRM detection scoped to known mobitool diagnostic LINES (Document is
  encrypted / DRM key not found / Invalid DRM pid / DRM expired / DRM support
  not included) -> no false-positive on book text; Print Replica -> clear fail.
- WithMemoryLimitPages cap; capped stdout/stderr writers; MaxOutputBytes.
- read-only fs.FS input mount + dedicated writable out dir; documented that
  FS isolation ultimately relies on running as a non-root user (memory-safety
  is the WASM boundary). validateEpub now requires STORED mimetype + verifies
  the container.xml OPF rootfile exists. Atomic moveFile. Closed-guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* ebookconvert: wire Kindle->EPUB into the read handler + capability endpoint

Server now transparently serves Kindle-family ebooks as EPUB when the admin
flag ebook.kindle_conversion_enabled is on and the WASM converter initialized.

- handlers.EbookConversion (converter + per-request flag predicate) on the read
  handler; HandleReadFile -> h.serveEbook. Kindle + enabled -> cached EPUB with
  X-Silo-Ebook-Conversion: converted, epub MIME, ETag = exact conversion cache
  key, must-revalidate. Failure (DRM/corrupt/oversize/unservable) -> raw
  original + X-Silo-Ebook-Conversion: failed + no-store, so the client opens
  externally. Context cancel propagates (not a conversion verdict).
- GET /api/v1/ebooks/capability advertises {enabled, source_formats,
  served_format, header contract}; enabled only when flag on AND converter
  wired (double gate) so the Android client can decide whether to flip
  mobi/azw/azw3 to in-app.
- router: buildEbookConversion compiles the module once at startup (feature off
  if it fails), cache dir is a sibling of TranscodeDir, flag read per request.

Codex review fixes folded in: ETag derived from the exact SourceKey cache key
(id+size+mtime+oshash+module version), not a weaker hash; no-store on the raw
fallback; open/stat failure of a produced EPUB falls back to raw per the
contract instead of 500. 10 handler tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(ebookconvert): harden conversion cache, HEAD path, and artifact verification

Addresses adversarial review + CodeRabbit findings on the Kindle->EPUB feature.

Correctness:
- Stop poisoning the negative cache on transient timeouts. Introduce
  ErrConversionTimedOut (distinct, non-wrapping ErrConversionFailed); classify
  the per-call timeout as transient and propagate a caller's cancel/deadline
  verbatim instead of reclassifying it as a conversion failure. remember() now
  only caches deterministic verdicts (DRM / failed), so a one-off timeout under
  load no longer wedges a convertible book onto raw-fallback for 6h.
- Detach the singleflight conversion from any single caller's context (DoChan +
  context.WithoutCancel), so one caller cancelling no longer aborts the shared
  work for the others; the cache is still populated for the next reader.
- enforceBudget never evicts the entry it is about to return, and skips other
  conversions' in-flight "converting-*" temp files.
- Cache hits refresh mtime so the mtime-ordered budget eviction is a real LRU,
  not FIFO.

Read path:
- HEAD is now cache-only via Cache.Lookup: a hit serves real converted headers,
  a negatively-cached source serves the failed contract, a miss advertises the
  converted representation cheaply without triggering a (minute-long, ~1 GiB)
  conversion. The GET still delivers the body + authoritative verdict.
- The admin flag is read through a short-TTL predicate so the read path and the
  capability endpoint no longer hit the DB per request.

Artifact / build:
- Add an in-code provenance test (embedded mobitool.wasm matches its recorded
  sha256) and a self-hosted CI job that runs the ebookconvert smoke conversions
  + provenance check, so the committed wasm can't silently rot.
- Pin + checksum-verify wasmtime in the build Dockerfile (drop curl|bash).

Docs: correct the design doc cache-key + setting-name descriptions, document the
HEAD/timeout/LRU semantics and resource limits, note DRM-marker brittleness, and
fix the README markdown table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* ci: remove ebookconvert workflow

---------

Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
2026-06-17 13:09:55 -04:00

112 lines
4.6 KiB
AMPL
Raw Blame History

module github.com/Silo-Server/silo-server
go 1.26.4
require (
github.com/aws/aws-sdk-go-v2 v1.41.5
github.com/aws/aws-sdk-go-v2/credentials v1.19.10
github.com/aws/aws-sdk-go-v2/service/s3 v1.97.3
github.com/go-chi/chi/v5 v5.2.5
github.com/golang-jwt/jwt/v5 v5.3.1
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.9.2
github.com/mattn/go-sqlite3 v1.14.34
github.com/prometheus/client_golang v1.23.2
github.com/prometheus/client_model v0.6.2 // indirect
github.com/redis/go-redis/v9 v9.18.0
golang.org/x/crypto v0.52.0
golang.org/x/sys v0.45.0
gopkg.in/yaml.v3 v3.0.1
)
require (
github.com/SherClockHolmes/webpush-go v1.4.0
github.com/abadojack/whatlanggo v1.0.1
github.com/go-chi/cors v1.2.2
github.com/gorilla/websocket v1.5.3
github.com/h2non/bimg v1.1.9
github.com/hashicorp/go-hclog v1.6.3
github.com/joho/godotenv v1.5.1
github.com/mmcdole/gofeed v1.3.0
github.com/oklog/ulid/v2 v2.1.0
github.com/pgvector/pgvector-go v0.3.0
github.com/pressly/goose/v3 v3.27.1
github.com/tetratelabs/wazero v1.12.0
github.com/wneessen/go-mail v0.7.3
github.com/zishang520/socket.io/v2 v2.5.0
go.n16f.net/thumbhash v1.1.0
golang.org/x/image v0.41.0
golang.org/x/net v0.55.0
google.golang.org/genproto/googleapis/rpc v0.0.0-20260420184626-e10c466a9529
gopkg.in/natefinch/lumberjack.v2 v2.2.1
)
require (
github.com/PuerkitoBio/goquery v1.8.0 // indirect
github.com/andybalholm/brotli v1.2.1 // indirect
github.com/andybalholm/cascadia v1.3.1 // indirect
github.com/dunglas/httpsfv v1.1.0 // indirect
github.com/fatih/color v1.13.0 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/gookit/color v1.5.4 // indirect
github.com/hashicorp/yamux v0.1.2 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/mattn/go-colorable v0.1.12 // indirect
github.com/mattn/go-isatty v0.0.21 // indirect
github.com/mfridman/interpolate v0.0.2 // indirect
github.com/mmcdole/goxpp v1.1.1-0.20240225020742-a0c311522b23 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/oklog/run v1.1.0 // indirect
github.com/quic-go/qpack v0.6.0 // indirect
github.com/quic-go/quic-go v0.60.0 // indirect
github.com/quic-go/webtransport-go v0.10.0 // indirect
github.com/rogpeppe/go-internal v1.14.1 // indirect
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 // indirect
github.com/sethvargo/go-retry v0.3.0 // indirect
github.com/vmihailenco/msgpack/v5 v5.4.1 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/xo/terminfo v0.0.0-20210125001918-ca9a967f8778 // indirect
github.com/zishang520/engine.io-go-parser v1.3.2 // indirect
github.com/zishang520/engine.io/v2 v2.5.0 // indirect
github.com/zishang520/socket.io-go-parser/v2 v2.5.0 // indirect
github.com/zishang520/webtransport-go v0.9.1 // indirect
go.opentelemetry.io/otel/sdk/metric v1.41.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
)
require (
github.com/Silo-Server/silo-plugin-sdk v0.7.0
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.8 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.21 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.22 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.7 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.13 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.21 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.21 // indirect
github.com/aws/smithy-go v1.24.2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/hashicorp/go-plugin v1.7.0
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/prometheus/common v0.66.1 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/robfig/cron/v3 v3.0.1
github.com/sony/sonyflake v1.3.0
go.uber.org/atomic v1.11.0 // indirect
go.yaml.in/yaml/v2 v2.4.2 // indirect
golang.org/x/sync v0.20.0
golang.org/x/text v0.37.0
golang.org/x/time v0.14.0
google.golang.org/grpc v1.80.0
google.golang.org/protobuf v1.36.11
)
replace github.com/zishang520/webtransport-go => ./internal/compat/zishang520-webtransport-go
Reference in New Issue View Git Blame Copy Permalink