silo-server/docs/design/2026-07-19-client-diagnostics.md

Client Diagnostics: crash reports and debug log upload

Status: Draft (spec + cross-repo plan) Date: 2026-07-19 Repos affected: silo-server, silo-android, silo-apple

Summary

Add an opt-in, self-hosted diagnostics pipeline: Silo clients detect crashes and abnormal exits, collect structured debug logs (playback, TV focus, network, lifecycle) plus a hardware snapshot (device model, display/HDMI, audio path, codec capabilities), and — only with explicit user consent — upload a compressed diagnostics bundle to the user's own Silo server. Admins review bundles through new acting-admin API endpoints alongside the existing server logs tooling.

Everything ships disabled by default:

• Server: a diagnostics.uploads_enabled server setting gates the whole feature (default off), and it cannot be enabled unless the server's private object storage is configured and validated.
• Client: verbose "debug logging" is a per-device setting (default off). Crash reporting defaults to Ask — after an abnormal exit the app shows an incident-specific prompt (e.g. "Silo closed unexpectedly during playback. Send a diagnostic report to your server?") with Send / Don't send / Always send. Nothing uploads without one of: an explicit prompt confirmation, a prior "Always send" election bound to this server and account, or the user explicitly sending a manual capture.

Motivation

Today we are blind to client-side failures:

• No crash reporting anywhere. Neither the Android repo nor the Apple repo installs an uncaught-exception handler, uses MetricKit, or bundles a crash SDK (verified: no handler/SDK references in either repo; the only Firebase dependency is messaging). A user saying "the app crashed" is a dead end.
• Diagnostics don't survive to the report. Android logs go to logcat (raw android.util.Log in ~40 files, no buffer or file sink); the Apple player's rich [CMP]/[CMP-DIAG] diagnostics go through print() (PlayerLog.swift / PlayerCore.swift) with no app-owned persistent sink. By the time a user reports a problem, the evidence is gone.
• The hardest bugs are device-specific. Shield HLG quirks, Fire TV passthrough suppression, HDMI mode switches, TV focus traps — reproducing them requires knowing the exact device, panel, audio chain, and codec path. Users can't reliably report any of that.
• The server already has half the story. operational_logs records playback_session_id (migration 028_log_partitioning.sql), the admin logs API filters by it, and the web admin has a logs viewer. What's missing is the client half, and a way to join the two.

Goals

• Capture crashes and abnormal exits (JVM/native/ANR on Android; crash and hang diagnostics on iOS; abnormal-exit detection on tvOS) with enough context to act on: stack or exit trace where the platform provides one, recent structured logs, and a device snapshot.
• Capture deep opt-in debug logs for playback, TV focus behavior, networking, and navigation/lifecycle — everything needed to troubleshoot user issues.
• Always include a hardware snapshot: device (Shield / Fire TV / onn / phone), OS, app build, display and HDMI mode with HDR capabilities, audio output path and passthrough capabilities, decoder inventory — with honest unknown / not_collected states where a platform can't provide a field.
• One payload contract across Android and Apple; one server endpoint; one admin surface. The contract is a checked-in versioned JSON Schema with golden fixtures consumed by all three repos' CI.
• Explicit, server-bound consent for every byte uploaded; feature fully off by default on both server and client.
• Bounded cost: size caps, per-account quotas, concurrency limits, and server-side retention.

Non-goals (v1)

• Live/continuous log streaming from clients to the server.
• Screenshots or screen recordings in bundles.
• Native/tvOS crash stack capture via an in-process crash reporter (PLCrashReporter/KSCrash-style). tvOS v1 detects abnormal exits and attaches breadcrumbs, not stacks (see Apple design). Adopting an in-process crash reporter is a separate decision with its own signal-safety and review implications.
• Symbolication. Bundles store what the OS gives us; a server-side symbolication job can come later.
• Native-app admin viewer screens (per current product policy, admin surfaces beyond STATS on the clients are a separate product decision; v1 admin consumption is via API + web admin).
• macOS. The Apple repo has a macOS target, but v1 scopes to android, android-tv, ios, tvos; the platform enum reserves macos without accepting it.
• Third-party telemetry, metrics aggregation, or fleet analytics. This is a support/debugging tool, not analytics.

Why this route

Upload to the user's own Silo server, not a third-party crash service. Crashlytics/Sentry SaaS would send user data to a third party, require vendor accounts/keys we'd have to ship, and put the reports in front of us rather than the person who can actually act — the self-hosting admin. Requiring every admin to also run self-hosted Sentry is far too heavy. The Silo server is the natural, privacy-aligned destination, and it already has auth, admin tooling, object storage, and retention patterns to build on.

Bundle-on-event, not streaming. A bundle captured at the moment of failure (crash, or a user-triggered capture) contains exactly the context needed: recent structured logs, the playback stats timeline, and the hardware snapshot. Continuous streaming would cost bandwidth and privacy while mostly capturing noise, and would still miss pre-crash buffers.

Hybrid storage: Postgres row + object-store blob. Report metadata (a small manifest: device summary, truncated crash summary, type, timestamps) goes in a queryable table; the payload itself (potentially megabytes of logs and crash artifacts) goes to the S3-compatible private bucket as a single archive. Ingesting client log lines into the partitioned operational_logs table was considered and rejected: bundles are consumed as whole documents by a human, not queried row-wise; client lines are untrusted input with different retention needs; and multi-MB uploads would bloat a table tuned for the server's own high-churn logs. The server does emit normalized operational events about diagnostics activity (report accepted / rejected / downloaded / deleted, with report id, account, sizes, result) so ingest and admin actions are auditable and correlate with existing logs — raw client lines never enter operational_logs.

A versioned archive, not a bare gzipped log stream. Crash evidence is heterogeneous: JSONL logs, ANR trace text, Android native tombstones (protobuf on API 31+), MetricKit diagnostic JSON. A tar.gz archive with named entries carries each artifact losslessly with a declared media type, keeps large traces out of Postgres, and can evolve by adding entries.

Build small in-house rather than adopt ACRA/other SDKs. The surface we need on Android (one exception handler writing a bounded marker, a ring buffer, one upload call) is a few hundred lines; ACRA would add a dependency and its own report format without covering Apple. On Apple, the OS provides MetricKit natively (iOS; not tvOS — see Apple design).

Architecture overview

 Android / Apple client                          silo-server
┌────────────────────────────────┐   ┌──────────────────────────────────────┐
│ Safe-logging facade → ring     │   │ GET  /api/v1/diagnostics/status      │
│  └ debug mode: rotating        │   │   └ available|disabled|              │
│    segment files               │   │     storage_unavailable + limits     │
│ Crash/abnormal-exit capture    │   │ POST /api/v1/diagnostics/reports     │
│  ├ UEH marker (Android)        │──▶│   ├ gate + storage check             │
│  ├ ApplicationExitInfo (30+)   │   │   ├ streaming, bounded multipart     │
│  ├ MetricKit (iOS)             │   │   ├ account quotas + concurrency     │
│  └ exit sentinel (tvOS)        │   │   ├ manifest → client_diagnostic_    │
│ Pending reports (server-bound) │   │   │   reports (state machine)        │
│ DeviceSnapshot (probes+new     │   │   └ archive → S3 private bucket      │
│   accessors)                   │   │ Admin (acting admin): list/detail/   │
│ Consent UI (ask/always/never,  │   │   download/delete + audit events     │
│   per server+account)          │   │ Retention + orphan reconciler task   │
└────────────────────────────────┘   └──────────────────────────────────────┘

Consent model

Consent is scoped, versioned, and destination-bound — not a global boolean. Both clients support multiple servers and accounts, so a device-wide "Always send" would let a bundle full of server A's titles and URLs upload to server B after a switch. Rules:

• Consent (including "Always send") is account-scoped: every consent record and every pending report is bound to (server_instance_id, account_user_id, consent_notice_version), where server_instance_id is a stable opaque ID returned by the status endpoint. The capturing profile_id is recorded on reports for attribution, not as a consent scope — quotas and elections follow the account.
• A pending report is only ever uploaded to the server+account it was captured under. Reports are never retargeted. Switching server or account neither uploads nor migrates pending reports; signing out of an account or removing a server deletes its pending reports.
• "Always send" is an election for one server+account. Bumping consent_notice_version (when we materially change what bundles contain) invalidates prior elections and reverts to Ask.
• Setting crash reports to Never deletes pending reports for that server+account and stops persistent capture (breadcrumbs/debug files); the in-memory ring keeps running (memory-only, never uploaded without a later explicit action).
• Child/managed profiles: the Diagnostics section is not shown to profiles with is_child, and every diagnostics action — changing settings, electing Always, the crash prompt, and manual capture/send — requires a non-child profile. A crash captured while a child profile was active is held pending and prompted for the next time a non-child profile is active. Reports are attributed to account + capturing profile.
• The crash prompt and the manual flow both offer View report — a human-readable summary (incident, categories and line counts, device identity, destination server) before anything is sent.
• Uploads carry consent: {mode, notice_version} in the manifest so the server can reject stale-consent uploads outright.

Bundle format (shared contract)

An upload is multipart/form-data with exactly two parts, in order:

1. manifest — application/json, ≤ 64 KiB. Queryable summary; stored in Postgres.
2. bundle — application/gzip: a tar.gz archive, manifest.json first. Entry names come from a fixed allowlist, and each name implies its media type — there is no per-entry type metadata to drift:

manifest.json            (manifest minus the `archive` object; see hashing note)
device.json              (full device snapshot)
logs.jsonl               (structured log lines, newest-last)
crash/summary.json       (normalized crash info incl. provenance)
crash/stack.txt          (JVM stack / ANR trace, when present)
crash/tombstone.pb       (Android native tombstone, opaque bytes, API 31+)
crash/metrickit.json     (raw MetricKit diagnostic JSON, iOS)
breadcrumbs.jsonl        (persisted pre-crash context journal, when present)

Hashing note: archive.sha256 and the size fields in part 1 describe the finalized bundle part bytes as transmitted. Since the archive is hashed after it is built, the embedded manifest.json is the manifest without the archive object; the server verifies part 1's archive fields against the received blob. archive.bytes is the transmitted (gzip) size; archive.uncompressed_bytes is the total decompressed tar stream size, including tar headers, the end-of-archive marker, and record padding — i.e. the byte count between the client's tar writer and gzip writer, and what gzip -l reports. Standard end-of-archive zero padding (up to 64 KiB) is accepted; any other trailing data is rejected.

The contract lives in this repo as a versioned JSON Schema (docs/design/schemas/client-diagnostics/v1/) with canonical valid and invalid fixtures; Go, Kotlin, and Swift CI all validate against the same fixtures. Servers accept all schema versions they support and ignore unknown additive fields.

Manifest (part 1)

{
  "schema_version": 1,
  "report": {
    "type": "crash",              // crash | anr | native_crash | hang | abnormal_exit | manual
    "captured_at": "2026-07-19T18:22:31Z",
    "capture_session_id": "run_c7d…",   // UUID per app run, also stamped on log lines
    "app_version": "1.4.2",
    "app_build": "20841",
    "platform": "android-tv",     // android | android-tv | ios | tvos  (macos reserved)
    "os_version": "11 (API 30)",
    "profile_id": "prof_…"        // capturing profile, attribution only
  },
  "destination": { "server_instance_id": "srv_…" },   // consent binding; server rejects mismatches
  "consent": { "mode": "prompt", "notice_version": 1 },  // prompt | always | manual
  "crash": {                      // absent for manual
    "summary": "NullPointerException in PlaybackSessionManager.start",
    "stack_excerpt": "…truncated ≤ 8 KiB; full artifact in archive…",
    "thread": "main",
    "foreground": true,
    "source": "ueh",              // ueh | exit_info | metrickit | exit_sentinel
    "provenance": "pre_failure",  // pre_failure | post_restart | metric_reporting_period
    "occurred_at": "2026-07-19T18:22:31Z"   // best known; MetricKit reports carry period bounds
  },
  "device_summary": { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "os":"11", "form_factor":"tv" },
  "playback_session_ids": ["ps_9f2…"],
  "log_summary": { "lines": 3841, "bytes_gz": 412034, "dropped_lines": 12, "categories": ["playback","focus","network"], "debug_logging": true },
  "archive": { "entries": ["manifest.json","device.json","logs.jsonl","crash/summary.json","crash/stack.txt"], "bytes": 498112, "uncompressed_bytes": 3110400, "sha256": "…" }
}

Log line schema (logs.jsonl)

{"ts":"2026-07-19T18:22:29.412Z","run":"run_c7d…","lvl":"W","cat":"playback","tag":"AudioCapabilityMgr","msg":"passthrough suppressed: HDMI sink lost TrueHD","attrs":{"sink":"HDMI","fmt":"truehd"}}

• lvl: V|D|I|W|E. cat: playback, focus, network, lifecycle, browse, cast, download, crash, other.
• run (capture_session_id) distinguishes lines from prior app runs when breadcrumbs or debug files span sessions.
• Playback additionally emits periodic (~5s) player-stats snapshot lines ("cat":"playback","tag":"StatsSnapshot") built from the existing stats models — a timeline of decoder, resolution, HDR mode, bitrate, dropped frames, and audio underruns around the failure.

Safe logging contract (redaction happens at collection time)

Best-effort scrubbing at package time is not enough: today's client logs already emit full URLs with query strings, token suffixes, and raw exception strings (e.g. the Apple HTTPClient response logging and player URL prints; Android exception messages containing backend URLs). Therefore:

• The facade accepts typed attributes, not preformatted strings, for anything risky: URL values are normalized (userinfo, query, and fragment always stripped; path preserved), exceptions pass through a sanitizing formatter, and free-text msg is authored, not interpolated from network data.
• Attribute keys are normatively registered: a per-category key registry (checked in next to the JSON Schema) declares each key and its value type (scalar types plus a few enumerated units). Unregistered keys fail debug builds/CI and are dropped in production builds.
• Existing call sites are migrated curated, not mechanically — each migrated line is reviewed against the contract. Unmigrated Log.*/print call sites simply never enter bundles.
• Defense in depth at bundle time: values matching the account's current access/profile tokens are replaced before packaging.
• Golden leak-fixture tests (JWTs, query tokens, signed URLs, cookies, token fragments) run in all three repos' CI against the facade and the bundle builder.
• The network category logs method, templated path, status, duration; never headers, bodies, query strings, or full URLs.

Device snapshot (device.json)

Fields are tri-state where a platform can't provide them: a value, "unknown" (probe exists, couldn't determine), or "not_collected" (no probe on this platform). Every snapshot carries captured_at and provenance (pre_failure for snapshots persisted with the incident, post_restart for snapshots taken at bundle-build time).

{
  "captured_at": "2026-07-19T18:22:31Z",
  "provenance": "pre_failure",
  "identity":  { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "device":"mdarcy", "form_factor":"tv", "device_id":"…" },
  "display":   { "mode":"3840x2160@59.94", "modes_supported":[…], "hdr_types":["HDR10","HLG","DV"], "wide_gamut":true },
  "audio":     { "outputs":[{"type":"HDMI","encodings":["AC3","EAC3","TRUEHD"],"channels":8}], "passthrough":["AC3","EAC3","TRUEHD"], "suppressions":[…] },
  "video_codecs": [ { "mime":"video/hevc", "hw":true, "profiles":["Main10","DV P8"], "max":"4096x2160@60" } ],
  "network":   { "transport":"ethernet" }
}

Android sources: AndroidDeviceMetadataProvider (id/model/platform/app version), MediaCodecCapabilitiesProbe (codec + HDR decode caps), AudioCapabilityManager (aggregate passthrough). New accessors required (the probes exist but don't currently export everything): current + supported Display.Mode list (DisplayHdrProbe exposes HDR support only), exact identity fields (Build.DEVICE, form factor), AudioManager.getDevices() output enumeration, and a snapshot/export API on PassthroughSuppressionRegistry (its set is currently private).

Apple sources: AppleDeviceIdentity.current (persisted id, device name, platform) and ApplePlaybackV3Capabilities.snapshot() (codec/HDR/device context). Honesty constraints: the current outputSnapshot() helper is private, reports route UIDs/types rather than full output capabilities, and its passthrough payload is a placeholder — so Apple v1 reports audio as route metadata with passthrough: "unknown" until a real probe exists, and the private helper is refactored into an internal shared accessor so diagnostics can reuse it without duplicating route logic. Audio route UIDs are persistent peripheral identifiers and are one-way hashed before inclusion.

Playback session correlation

Clients include recent server playback session IDs so admins can pivot into the existing server-side logs (operational_logs.playback_session_id + the admin logs filter). Both clients currently retain only the active session ID (Android PlaybackSessionManager, Apple PlaybackSessionBridge), so each client adds a small bounded, persisted recent-session tracker (last ~10 session IDs with timestamps) that survives restarts — required for next-launch crash reports.

Server design (silo-server)

Feature gate

Borrows the behavior of the requests gate (status endpoint + per-call guard that rereads settings) but stores keys in the general server_settings store — requests uses its own dedicated table, which diagnostics doesn't need. Key naming follows existing namespaces (opslog.*, audiobooks.enabled, playback.protocol_v3_enabled):

• diagnostics.uploads_enabled (bool, default false)
• diagnostics.max_bundle_bytes (default 10 MiB compressed)
• diagnostics.max_uncompressed_bytes (default 64 MiB)
• diagnostics.max_reports_per_user_per_day (default 20)
• diagnostics.retention_days (default 30)
• diagnostics.max_bytes_per_user (default 200 MiB)

Enabling diagnostics.uploads_enabled requires a storage validation (private object store configured and writable); the admin settings API refuses to enable it otherwise. S3Private is a nullable dependency in the router and is only populated when configured, so the feature must degrade predictably, not 500.

Status endpoint

GET /api/v1/diagnostics/status — authenticated (RequireAuth), no profile required: crashes during login/profile selection must still be reportable, so capability discovery and upload are account-scoped with optional profile attribution.

{
  "status": "available",            // available | disabled | storage_unavailable
  "server_instance_id": "srv_…",    // stable; binds consent + pending reports
  "accepted_schema_versions": [1],
  "max_bundle_bytes": 10485760,
  "max_manifest_bytes": 65536,
  "retention_days": 30,
  "consent_notice_version": 1
}

Clients cache this alongside existing server-driven config refresh.

Ingest endpoint

POST /api/v1/diagnostics/reports — RequireAuth; access-token Authorization header only (no query credentials; sa_ API keys rejected for diagnostics). Optional validated profile attribution. Hardened parsing — the avatar handler is the precedent (multipart + PutObject), but its buffer-everything approach (ParseMultipartForm + io.ReadAll) is not reused:

1. Guard: diagnostics.uploads_enabled + storage present, else stable error codes (disabled, storage_unavailable).
2. http.MaxBytesReader with max_bundle_bytes + manifest overhead before any parsing; request deadline; per-account concurrency cap (1 in-flight) and a server-global concurrency ceiling, on top of the existing rate-limit middleware.
3. Streaming multipart.Reader: exactly the two named parts in order; manifest capped at 64 KiB and schema-validated; unknown parts rejected.
4. Quota reservation: in one transaction holding a per-account advisory lock (pg_advisory_xact_lock(user_id) — serializing concurrent uploads from multiple devices on one account), check the per-day count and per-user byte quota and insert the report row in state receiving (server-generated UUID + short_id). Failures return quota_exceeded (429, with retry_after) or too_large (413) — machine-readable so clients back off and keep the report pending locally.
5. Stream the bundle part directly to its final key (diagnostics/{user_id}/{report_id}.tar.gz) while hashing and counting bytes; enforce the compressed cap during streaming. Validate the gzip/tar structure and per-entry limits (entry count, name allowlist, uncompressed total ≤ max_uncompressed_bytes, compression ratio bound) in the same bounded streaming pass. Reject on violation (invalid_bundle, unsupported_schema). There is no staging/rename step — S3-style stores have no cheap rename, and the row's state is the source of truth: a blob is not a report until its row is ready.
6. On success, update the row to ready with sizes + sha256. On any failure: delete the object, mark the row failed (compensation); the reconciler (below) cleans up stragglers from process death mid-request.
7. Respond 201 {"report_id":"…","short_id":"SILO-…"} and emit a normalized operational event (accepted/rejected, account, sizes, result).

short_id is 12 Crockford-Base32 characters, server-generated with collision retry, case-insensitively unique, shown to the user for support conversations, filterable in the admin API — and explicitly not an authorization secret.

New code: internal/diagnostics/ (service + store interface), handler in internal/api/handlers/diagnostics.go, wired through Dependencies in internal/api/router.go. The store interface is diagnostics-owned (the avatar handler's is package-private and presign-only) and covers everything the design uses: a streaming put (io.Reader — the existing s3client PutObject takes []byte, so this is a small s3client addition), GetObject (streamed admin download fallback), DeleteObject, ListObjects(prefix) (reconciler), and PresignGetURL.

Storage

CREATE TABLE client_diagnostic_reports (
  id            UUID PRIMARY KEY,
  short_id      TEXT NOT NULL,                 -- unique, case-insensitive (expression index)
  user_id       INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  profile_id    TEXT,                          -- capturing profile, nullable
  state         TEXT NOT NULL,                 -- receiving | ready | failed
  captured_at   TIMESTAMPTZ NOT NULL,          -- client claim
  received_at   TIMESTAMPTZ NOT NULL DEFAULT now(),  -- server truth; retention keys off this
  report_type   TEXT NOT NULL,                 -- crash|anr|native_crash|hang|abnormal_exit|manual
  platform      TEXT NOT NULL,
  app_version   TEXT NOT NULL,
  crash_summary TEXT,                          -- truncated ≤ 8 KiB
  manifest      JSONB NOT NULL,                -- part 1 verbatim (≤ 64 KiB by construction)
  playback_session_ids TEXT[] NOT NULL DEFAULT '{}',
  blob_bucket   TEXT,
  blob_key      TEXT,
  blob_bytes    BIGINT,
  uncompressed_bytes BIGINT,
  blob_sha256   TEXT
);
CREATE UNIQUE INDEX ON client_diagnostic_reports (lower(short_id));
CREATE INDEX ON client_diagnostic_reports (user_id, received_at DESC);
CREATE INDEX ON client_diagnostic_reports (received_at);

Plain table, not partitioned — volume is orders of magnitude below operational_logs; the byte-heavy payload lives in object storage. Full stacks/traces live only in the archive; the row holds a truncated summary.

Admin API

Under the acting-admin subtree (same placement as the admin_logs.go routes):

• GET /api/v1/admin/diagnostics/reports — list; filters: user, platform, type, date range, short_id (exact, case-insensitive); paginated.
• GET /api/v1/admin/diagnostics/reports/{id} — row incl. manifest.
• GET /api/v1/admin/diagnostics/reports/{id}/download — presigned GET URL (primary, matching avatar behavior) with a streamed GetObject proxy fallback for deployments where presigned URLs aren't reachable — the fallback is new behavior, not something the avatar path provides.
• DELETE /api/v1/admin/diagnostics/reports/{id} — row + blob.

Admin download and delete emit audit/operational events. Web admin gets a list/detail/download page next to the logs viewer, with a playback-session-id link into the existing filtered logs view. Native apps add no admin screens in v1 (product policy).

Retention + reconciliation

A taskmanager task internal/taskmanager/tasks/cleanup_client_diagnostics.go mirroring cleanup_operational_log.go, registered in cmd/silo/main.go with the other cleanup tasks:

• Delete reports older than diagnostics.retention_days (by received_at), then enforce diagnostics.max_bytes_per_user oldest-first. Blob deleted first (tolerating already-missing objects), then the row; deletes are idempotent.
• Reconcile: remove receiving/failed rows older than a grace window, and (via ListObjects) any diagnostics/* objects whose row is absent, failed, or stale-receiving — so partial failures and process deaths mid-request never leak storage.
• Account deletion cascades rows (FK) ; the reconciler then removes orphaned blobs.

Android design (silo-android)

New package android-shared/src/androidMain/kotlin/org/siloserver/silo/common/diagnostics/.

Logging

• SiloLog.kt — safe-logging facade per the contract above: SiloLog.d(cat, tag, msg, attrs) (+i/w/e, sanitized Throwable). Always forwards to android.util.Log (today's behavior unchanged) and offers the entry to the ring buffer.
• LogRing.kt — always-on in-memory ring: fixed-size array of pre-rendered compact entries, non-blocking writes (atomic index; a torn final entry during crash snapshot is acceptable), ~4000 entries / ~1.5 MB cap, lazy attribute formatting off the hot path, dropped-entry counter. Must add no measurable contention to playback; a playback benchmark gate accompanies the PR that wires player events in.
• DiagnosticsFileLogger.kt — only while debug logging is on: a single-writer coroutine draining a bounded channel (drop + count under backpressure) into append-only framed segment files (5 × 2 MB, app-private files/diagnostics/logs/, noBackup), compressed only at bundle-build time — a process death never corrupts more than the tail of the active segment (unlike a single long-lived gzip stream).
• Playback wiring: each PlaybackAnalyticsListener event plus a ~5s periodic PlayerStatsSnapshot line while debug logging is on (reuses the existing reducer). TV focus logging hooks the TV app's focus observers under cat=focus. Network: a Ktor hook logging method, templated path, status, duration_ms only.

Crash and abnormal-exit capture

• CrashCapture.kt — installed as early as possible in both Application classes; idempotent; captures the previous Thread.getDefaultUncaughtExceptionHandler once. On crash it does the minimum on the dying thread: render the stack (bounded), copy what the non-blocking ring exposes within a hard time budget, write one bounded marker file (≤ ~512 KiB) via temp-write + atomic rename, and invoke the prior handler in a finally. No locks shared with the ring's write path, no coroutines, no allocation-heavy JSON building (preformatted line buffer). Bundle assembly happens on next launch, never in the handler. At most 3 pending reports are kept (oldest dropped) as a crash-loop guard.
• ExitInfoCollector.kt — API 30+ enhancement (minSdk is 24; many Fire TV devices ship below API 30, where the handler is the only source): reads ActivityManager.getHistoricalProcessExitReasons() on launch. REASON_CRASH, REASON_CRASH_NATIVE, REASON_ANR become pending reports. Dedupe uses a persisted bounded fingerprint set (process name, pid, timestamp, reason, status, trace hash) — not a raw timestamp watermark, which loses same-stamp events and breaks on clock changes. getTraceInputStream() is treated as optional opaque bytes with a bounded read: ANR text → crash/stack.txt; API 31+ native tombstone protobuf → crash/tombstone.pb (never decoded into the manifest). UEH-captured events are correlated by fingerprint so the same death isn't reported twice.
• Next foreground launch, per the consent model: Ask → aggregated, incident-specific prompt (one prompt covering N pending reports; default focus Don't send on TV; "Always send" requires a second confirmation; full-screen review flow on TV rather than a small dialog); Always → silent upload, at most one bundle per crash fingerprint per day; Never → purge. Per-fingerprint prompt suppression with backoff prevents crash-loop prompt fatigue. Pending reports expire after 7 days — visibly (the Diagnostics screen shows pending reports with expiry), not silently.

Device snapshot

DeviceSnapshotCollector.kt assembles device.json from the existing probes plus the new accessors listed in the contract section (display mode list, Build.DEVICE/form factor, AudioManager.getDevices() enumeration, suppression-registry export). A compact snapshot is persisted with each crash marker (provenance: pre_failure) since post-restart audio/display state can differ from crash-time state.

Settings + consent UI

• DiagnosticsSettingsStore.kt — DataStore. Deliberately not the per-profile pattern of AndroidPlayerSettingsStore: consent and pending reports are keyed by server+account (per the consent model), and debug logging is per-device. Holds consent records, crash-report mode, debug-logging flag, fingerprint set, session tracker, and status cache.
• Phone: "Diagnostics" section in SettingsScreen.kt; TV: rail category in TvSettingsScreen.kt. For non-child profiles the section is always visible and state-aware rather than hidden when the server gate is off — it distinguishes available / disabled by server / storage unavailable / offline, and always shows local pending reports (count, size, expiry) with review/delete controls. Uploads are simply unavailable in non-available states; a report captured while the gate was off is not auto-uploaded later unless consent for that server+account exists.
• Manual capture is the primary support flow and is stronger than a bare "send now": Start diagnostic capture (turns on verbose capture, shows an indicator) → user reproduces the issue → Stop & review → summary (categories, time range, sizes, destination) → send. A one-shot "Send diagnostics now" remains for quick cases but warns when only the in-memory ring is available ("this bundle contains only the last few minutes of basic logs").
• After upload, the short_id is displayed and kept in a small sent history so a user can read it to their admin later.

Upload

• shared/commonMain/…/network/api/DiagnosticsApi.kt — getStatus() and uploadReport(manifestJson, bundleBytes) via Ktor MultiPartFormDataContent (first multipart use in the client — verified none exists — written as a small reusable helper), using the existing safeApiCall/ApiResult pattern, wired through Koin like other *Api classes.
• DiagnosticsUploader.kt — builds the tar.gz (marker + ring/segment files overlapping the incident window + breadcrumbs + snapshot), streams it to the endpoint, deletes the pending report only on 201, records short_id. On quota_exceeded/too_large/disabled/storage_unavailable it keeps the pending report and backs off per retry_after; on unsupported_schema it surfaces "server needs an update" rather than silently retrying until expiry.

Apple design (silo-apple)

Mirrors Android with platform-native mechanisms, and with tvOS scoped honestly — MetricKit diagnostic payloads (crash/hang) are iOS-only; the SDK marks them unavailable on tvOS, and delivery is delayed (typically daily, while the app runs), not "next launch".

• Ring + facade: DiagLog implementing the same safe-logging contract. The player's existing cmpLog helper (PlayerLog.swift) is extended into a tee — it keeps print()ing (preserving devicectl --console capture on tvOS) and also feeds the ring; remaining direct print() call sites in Screens/Player/ migrate to it curated, per the redaction contract. OSLog-based subsystems are harvested at bundle-build time via OSLogStore(scope: .currentProcessIdentifier) (deployment targets are iOS/tvOS 26, far above the OSLogStore 15+ floor), merged into logs.jsonl with provenance.
• Breadcrumb journal: because MetricKit reports arrive in a later process, the in-memory ring is gone by delivery time. While crash reporting is enabled, a small append-only breadcrumb journal (bounded, rotating, crash-safe appends: lifecycle transitions, playback start/stop, route changes, screen changes) persists across runs and becomes breadcrumbs.jsonl, matched to a diagnostic by capture_session_id and time window. Snapshots taken at bundle-build time are labeled provenance: post_restart; the design never presents post-restart state as crash-time state.
• iOS crash/hang detection: an MXMetricManager subscriber. Each received MXCrashDiagnostic/MXHangDiagnostic becomes its own pending report: raw diagnostic JSON preserved as crash/metrickit.json, deduped by a canonical hash of the diagnostic (not timestamps), carrying its own app version and reporting-period bounds. Prompt copy is type-specific ("crashed" vs "was not responding") and honest about timing ("yesterday during playback"). Delivery is verified on physical devices; parsing is tested via injected fixtures.
• tvOS abnormal-exit detection: no crash stacks in v1. A dirty-exit sentinel (marker written at launch, cleared on clean background/termination) plus the breadcrumb journal yields type: abnormal_exit reports — "Silo did not shut down cleanly last time" — with breadcrumbs and device snapshot but no stack. An in-process crash reporter for tvOS is explicitly deferred (see Non-goals).
• Device snapshot: AppleDeviceIdentity.current + ApplePlaybackV3Capabilities.snapshot() for identity/codec/HDR context; audio reported as route metadata with hashed route UIDs and passthrough: "unknown" until a real output-capability probe exists (the current one is a private placeholder).
• Settings/consent: Diagnostics section in SettingsView (iOS) and TVSettingsView (tvOS) with the same state-aware presentation, manual capture flow, and consent rules.
• Upload: a small multipart body writer added to HTTPClient (verified: JSON-only today, no multipart/uploadTask), same status/error handling as Android.
• Housekeeping note: product bundle IDs are already org.siloserver.silo; only legacy keychain/logging identifiers (com.continuum.*) remain, so nothing here blocks on rebranding.

Privacy summary

• Consent boundaries: no network transmission of diagnostic data without prompt confirmation, a server+account-bound Always election, or an explicit manual send. The always-on ring is memory-only; persistent capture (segments, breadcrumbs, markers) is app-private, size-capped, backup-excluded, and purged by Never/sign-out/server removal.
• Never collected: credentials, Authorization/profile-token values, request bodies, query strings, full URLs (normalized at collection), unsanitized exception dumps. Bearer-token matching at bundle time as defense in depth. Golden leak fixtures in CI.
• Deliberately collected (disclosed in View report): device identity incl. device name, media titles appearing in playback logs, templated server paths. The recipient is the user's own server admin, who already has server-side access to this information; that is the trust model this feature is scoped to — and why third-party destinations are out.
• Persistent identifiers: the existing device IDs (already sent to the server today) plus hashed audio-route UIDs; each is enumerated in the schema docs with its purpose and retention.
• User control: state-aware Diagnostics screen with pending list, review, delete, sent history; admin-side deletion; account deletion cascades reports; admin access is acting-admin-only and audited.
• Bounded retention: defaults 30 days / 200 MiB per user, admin-tunable, enforced server-side.

Rollout plan (vertical slices, each independently shippable)

1. Server foundation — settings + storage-validated gate, status endpoint, hardened ingest with state machine, schema + fixtures, admin API, retention/reconciler task, operational audit events. Feature stays off; nothing user-visible.
2. Contract fixtures in clients — JSON Schema + golden fixtures wired into Android/Apple CI; safe-logging facades + rings land dark (no UI, no capture persistence, no prompts).
3. Android end-to-end slice — consent model + Diagnostics settings UI + crash marker/ExitInfo capture + device snapshot + uploader + manual capture flow, with the initial curated logging migration (player tags + TV focus + network hook + stats snapshots). First release where a user can be asked for, and send, a useful report.
4. Android coverage expansion — remaining curated call-site migration, breadcrumbs-on-Android if wanted, prompt-fatigue tuning from the field.
5. Apple slice — cmpLog tee + OSLogStore harvest + breadcrumbs + MetricKit (iOS) + exit sentinel (tvOS) + settings/consent + multipart upload.
6. Web admin viewer + correlation UX — list/detail/download page, playback-session pivot into the logs viewer.
7. Later / separate decisions — symbolication job, tvOS in-process crash reporter, native-app admin viewer, macOS.

Slices 3 and 5 ship complete consent→capture→upload flows; intermediate releases never prompt for reports they can't send.

Open questions

1. Should users be able to delete their uploaded reports from the client (self-service deletion), or is pending-report control + admin deletion enough for v1?
2. tvOS: is abnormal-exit detection (no stacks) acceptable for v1, or does the Shield/Fire TV debugging need justify evaluating an in-process crash reporter (PLCrashReporter/KSCrash) with its signal-safety and review trade-offs?
3. Do we want breadcrumbs on Android in v1 (cheap once the facade exists), or is the UEH marker + ExitInfo enough there?
4. Localization of prompt/consent strings at launch vs English-first.
5. Exact quota defaults (20/day, 200 MiB/user, 10 MiB/bundle) — tune before enabling anywhere real.
6. Whether the status payload should also fold into an existing server-driven config refresh response later (clients poll both today); the dedicated endpoint ships first either way.

Review notes

This spec was adversarially reviewed before the draft PR: two independent review passes (a fact-check of every codebase claim against the three repos, and a design attack across consent, crash-capture correctness, ingest robustness, storage consistency, UX, and phasing). Material corrections adopted include: tvOS MetricKit unavailability and honest MetricKit timing; server+account-bound consent; storage-validated gating (private object store is optional in real deployments); streaming bounded ingest with a report state machine and orphan reconciliation; the archive container replacing a bare log gzip; collection-time typed redaction replacing scrub-at-package; fingerprint-based ExitInfo dedupe; non-blocking ring + bounded crash marker; recent-session trackers (only the active playback session ID exists today); and honest Apple snapshot capabilities (passthrough currently unknown).