From 7369d0afd8d8b4007ea92939fed184fe717003fe Mon Sep 17 00:00:00 2001 From: Quick104 <31828688+Quick104@users.noreply.github.com> Date: Sun, 19 Jul 2026 16:27:37 -0400 Subject: [PATCH] docs(design): client diagnostics spec (crash reports + debug log upload) Cross-repo spec and rollout plan for opt-in client crash reporting and debug-log upload to the user's own Silo server: silo-server ingest, storage, admin API, and retention; silo-android and silo-apple capture, consent, and upload. Disabled by default on both server and client. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01XppCCycoaskCsW7ja1fZct --- docs/design/2026-07-19-client-diagnostics.md | 766 +++++++++++++++++++ 1 file changed, 766 insertions(+) create mode 100644 docs/design/2026-07-19-client-diagnostics.md diff --git a/docs/design/2026-07-19-client-diagnostics.md b/docs/design/2026-07-19-client-diagnostics.md new file mode 100644 index 00000000..4c929325 --- /dev/null +++ b/docs/design/2026-07-19-client-diagnostics.md @@ -0,0 +1,766 @@ +# Client Diagnostics: crash reports and debug log upload + +Status: Draft (spec + cross-repo plan) +Date: 2026-07-19 +Repos affected: `silo-server`, `silo-android`, `silo-apple` + +## Summary + +Add an opt-in, self-hosted diagnostics pipeline: Silo clients detect crashes +and abnormal exits, collect structured debug logs (playback, TV focus, +network, lifecycle) plus a hardware snapshot (device model, display/HDMI, +audio path, codec capabilities), and — only with explicit user consent — +upload a compressed diagnostics bundle to the user's own Silo server. Admins +review bundles through new acting-admin API endpoints alongside the existing +server logs tooling. + +Everything ships **disabled by default**: + +- Server: a `diagnostics.uploads_enabled` server setting gates the whole + feature (default off), and it cannot be enabled unless the server's private + object storage is configured and validated. +- Client: verbose "debug logging" is a per-device setting (default off). + Crash reporting defaults to **Ask** — after an abnormal exit the app shows + an incident-specific prompt (e.g. *"Silo closed unexpectedly during + playback. Send a diagnostic report to your server?"*) with + **Send / Don't send / Always send**. Nothing uploads without one of: an + explicit prompt confirmation, a prior "Always send" election bound to this + server and account, or the user explicitly sending a manual capture. + +## Motivation + +Today we are blind to client-side failures: + +- **No crash reporting anywhere.** Neither the Android repo nor the Apple + repo installs an uncaught-exception handler, uses MetricKit, or bundles a + crash SDK (verified: no handler/SDK references in either repo; the only + Firebase dependency is messaging). A user saying "the app crashed" is a + dead end. +- **Diagnostics don't survive to the report.** Android logs go to logcat (raw + `android.util.Log` in ~40 files, no buffer or file sink); the Apple + player's rich `[CMP]`/`[CMP-DIAG]` diagnostics go through `print()` + (`PlayerLog.swift` / `PlayerCore.swift`) with no app-owned persistent sink. + By the time a user reports a problem, the evidence is gone. +- **The hardest bugs are device-specific.** Shield HLG quirks, Fire TV + passthrough suppression, HDMI mode switches, TV focus traps — reproducing + them requires knowing the exact device, panel, audio chain, and codec path. + Users can't reliably report any of that. +- **The server already has half the story.** `operational_logs` records + `playback_session_id` (migration `028_log_partitioning.sql`), the admin + logs API filters by it, and the web admin has a logs viewer. What's missing + is the client half, and a way to join the two. + +## Goals + +- Capture crashes and abnormal exits (JVM/native/ANR on Android; crash and + hang diagnostics on iOS; abnormal-exit detection on tvOS) with enough + context to act on: stack or exit trace where the platform provides one, + recent structured logs, and a device snapshot. +- Capture deep opt-in debug logs for playback, TV focus behavior, networking, + and navigation/lifecycle — everything needed to troubleshoot user issues. +- Always include a hardware snapshot: device (Shield / Fire TV / onn / + phone), OS, app build, display and HDMI mode with HDR capabilities, audio + output path and passthrough capabilities, decoder inventory — with honest + `unknown` / `not_collected` states where a platform can't provide a field. +- One payload contract across Android and Apple; one server endpoint; one + admin surface. The contract is a checked-in versioned JSON Schema with + golden fixtures consumed by all three repos' CI. +- Explicit, server-bound consent for every byte uploaded; feature fully off + by default on both server and client. +- Bounded cost: size caps, per-account quotas, concurrency limits, and + server-side retention. + +## Non-goals (v1) + +- Live/continuous log streaming from clients to the server. +- Screenshots or screen recordings in bundles. +- Native/tvOS crash *stack* capture via an in-process crash reporter + (PLCrashReporter/KSCrash-style). tvOS v1 detects abnormal exits and + attaches breadcrumbs, not stacks (see Apple design). Adopting an in-process + crash reporter is a separate decision with its own signal-safety and review + implications. +- Symbolication. Bundles store what the OS gives us; a server-side + symbolication job can come later. +- Native-app admin viewer screens (per current product policy, admin surfaces + beyond STATS on the clients are a separate product decision; v1 admin + consumption is via API + web admin). +- macOS. The Apple repo has a macOS target, but v1 scopes to + `android`, `android-tv`, `ios`, `tvos`; the platform enum reserves `macos` + without accepting it. +- Third-party telemetry, metrics aggregation, or fleet analytics. This is a + support/debugging tool, not analytics. + +## Why this route + +**Upload to the user's own Silo server, not a third-party crash service.** +Crashlytics/Sentry SaaS would send user data to a third party, require vendor +accounts/keys we'd have to ship, and put the reports in front of *us* rather +than the person who can actually act — the self-hosting admin. Requiring +every admin to also run self-hosted Sentry is far too heavy. The Silo server +is the natural, privacy-aligned destination, and it already has auth, admin +tooling, object storage, and retention patterns to build on. + +**Bundle-on-event, not streaming.** A bundle captured at the moment of +failure (crash, or a user-triggered capture) contains exactly the context +needed: recent structured logs, the playback stats timeline, and the hardware +snapshot. Continuous streaming would cost bandwidth and privacy while mostly +capturing noise, and would still miss pre-crash buffers. + +**Hybrid storage: Postgres row + object-store blob.** Report *metadata* (a +small manifest: device summary, truncated crash summary, type, timestamps) +goes in a queryable table; the payload itself (potentially megabytes of logs +and crash artifacts) goes to the S3-compatible private bucket as a single +archive. Ingesting client log lines into the partitioned `operational_logs` +table was considered and rejected: bundles are consumed as whole documents by +a human, not queried row-wise; client lines are untrusted input with +different retention needs; and multi-MB uploads would bloat a table tuned for +the server's own high-churn logs. The server *does* emit normalized +operational events about diagnostics activity (report accepted / rejected / +downloaded / deleted, with report id, account, sizes, result) so ingest and +admin actions are auditable and correlate with existing logs — raw client +lines never enter `operational_logs`. + +**A versioned archive, not a bare gzipped log stream.** Crash evidence is +heterogeneous: JSONL logs, ANR trace text, Android native tombstones +(protobuf on API 31+), MetricKit diagnostic JSON. A tar.gz archive with named +entries carries each artifact losslessly with a declared media type, keeps +large traces out of Postgres, and can evolve by adding entries. + +**Build small in-house rather than adopt ACRA/other SDKs.** The surface we +need on Android (one exception handler writing a bounded marker, a ring +buffer, one upload call) is a few hundred lines; ACRA would add a dependency +and its own report format without covering Apple. On Apple, the OS provides +MetricKit natively (iOS; not tvOS — see Apple design). + +## Architecture overview + +``` + Android / Apple client silo-server +┌────────────────────────────────┐ ┌──────────────────────────────────────┐ +│ Safe-logging facade → ring │ │ GET /api/v1/diagnostics/status │ +│ └ debug mode: rotating │ │ └ available|disabled| │ +│ segment files │ │ storage_unavailable + limits │ +│ Crash/abnormal-exit capture │ │ POST /api/v1/diagnostics/reports │ +│ ├ UEH marker (Android) │──▶│ ├ gate + storage check │ +│ ├ ApplicationExitInfo (30+) │ │ ├ streaming, bounded multipart │ +│ ├ MetricKit (iOS) │ │ ├ account quotas + concurrency │ +│ └ exit sentinel (tvOS) │ │ ├ manifest → client_diagnostic_ │ +│ Pending reports (server-bound) │ │ │ reports (state machine) │ +│ DeviceSnapshot (probes+new │ │ └ archive → S3 private bucket │ +│ accessors) │ │ Admin (acting admin): list/detail/ │ +│ Consent UI (ask/always/never, │ │ download/delete + audit events │ +│ per server+account) │ │ Retention + orphan reconciler task │ +└────────────────────────────────┘ └──────────────────────────────────────┘ +``` + +## Consent model + +Consent is **scoped, versioned, and destination-bound** — not a global +boolean. Both clients support multiple servers and accounts, so a device-wide +"Always send" would let a bundle full of server A's titles and URLs upload to +server B after a switch. Rules: + +- Consent (including "Always send") is **account-scoped**: every consent + record and every pending report is bound to + `(server_instance_id, account_user_id, consent_notice_version)`, where + `server_instance_id` is a stable opaque ID returned by the status + endpoint. The capturing `profile_id` is recorded on reports for + *attribution*, not as a consent scope — quotas and elections follow the + account. +- A pending report is only ever uploaded to the server+account it was + captured under. Reports are never retargeted. Switching server or account + neither uploads nor migrates pending reports; signing out of an account or + removing a server deletes its pending reports. +- "Always send" is an election for one server+account. Bumping + `consent_notice_version` (when we materially change what bundles contain) + invalidates prior elections and reverts to Ask. +- Setting crash reports to **Never** deletes pending reports for that + server+account and stops persistent capture (breadcrumbs/debug files); the + in-memory ring keeps running (memory-only, never uploaded without a later + explicit action). +- **Child/managed profiles**: the Diagnostics section is not shown to + profiles with `is_child`, and every diagnostics action — changing + settings, electing Always, the crash prompt, and manual capture/send — + requires a non-child profile. A crash captured while a child profile was + active is held pending and prompted for the next time a non-child profile + is active. Reports are attributed to account + capturing profile. +- The crash prompt and the manual flow both offer **View report** — a + human-readable summary (incident, categories and line counts, device + identity, destination server) before anything is sent. +- Uploads carry `consent: {mode, notice_version}` in the manifest so the + server can reject stale-consent uploads outright. + +## Bundle format (shared contract) + +An upload is `multipart/form-data` with exactly two parts, in order: + +1. `manifest` — `application/json`, ≤ 64 KiB. Queryable summary; stored in + Postgres. +2. `bundle` — `application/gzip`: a tar.gz archive, `manifest.json` first. + Entry names come from a **fixed allowlist**, and each name implies its + media type — there is no per-entry type metadata to drift: + +``` +manifest.json (manifest minus the `archive` object; see hashing note) +device.json (full device snapshot) +logs.jsonl (structured log lines, newest-last) +crash/summary.json (normalized crash info incl. provenance) +crash/stack.txt (JVM stack / ANR trace, when present) +crash/tombstone.pb (Android native tombstone, opaque bytes, API 31+) +crash/metrickit.json (raw MetricKit diagnostic JSON, iOS) +breadcrumbs.jsonl (persisted pre-crash context journal, when present) +``` + +Hashing note: `archive.sha256` and the size fields in part 1 describe the +finalized `bundle` part bytes as transmitted. Since the archive is hashed +after it is built, the embedded `manifest.json` is the manifest **without** +the `archive` object; the server verifies part 1's `archive` fields against +the received blob. + +The contract lives in this repo as a versioned JSON Schema +(`docs/design/schemas/client-diagnostics/v1/`) with canonical valid and +invalid fixtures; Go, Kotlin, and Swift CI all validate against the same +fixtures. Servers accept all schema versions they support and ignore unknown +additive fields. + +### Manifest (part 1) + +```json +{ + "schema_version": 1, + "report": { + "type": "crash", // crash | anr | native_crash | hang | abnormal_exit | manual + "captured_at": "2026-07-19T18:22:31Z", + "capture_session_id": "run_c7d…", // UUID per app run, also stamped on log lines + "app_version": "1.4.2", + "app_build": "20841", + "platform": "android-tv", // android | android-tv | ios | tvos (macos reserved) + "os_version": "11 (API 30)", + "profile_id": "prof_…" // capturing profile, attribution only + }, + "destination": { "server_instance_id": "srv_…" }, // consent binding; server rejects mismatches + "consent": { "mode": "prompt", "notice_version": 1 }, // prompt | always | manual + "crash": { // absent for manual + "summary": "NullPointerException in PlaybackSessionManager.start", + "stack_excerpt": "…truncated ≤ 8 KiB; full artifact in archive…", + "thread": "main", + "foreground": true, + "source": "ueh", // ueh | exit_info | metrickit | exit_sentinel + "provenance": "pre_failure", // pre_failure | post_restart | metric_reporting_period + "occurred_at": "2026-07-19T18:22:31Z" // best known; MetricKit reports carry period bounds + }, + "device_summary": { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "os":"11", "form_factor":"tv" }, + "playback_session_ids": ["ps_9f2…"], + "log_summary": { "lines": 3841, "bytes_gz": 412034, "dropped_lines": 12, "categories": ["playback","focus","network"], "debug_logging": true }, + "archive": { "entries": ["manifest.json","device.json","logs.jsonl","crash/summary.json","crash/stack.txt"], "bytes": 498112, "uncompressed_bytes": 3110400, "sha256": "…" } +} +``` + +### Log line schema (`logs.jsonl`) + +```json +{"ts":"2026-07-19T18:22:29.412Z","run":"run_c7d…","lvl":"W","cat":"playback","tag":"AudioCapabilityMgr","msg":"passthrough suppressed: HDMI sink lost TrueHD","attrs":{"sink":"HDMI","fmt":"truehd"}} +``` + +- `lvl`: `V|D|I|W|E`. `cat`: `playback`, `focus`, `network`, `lifecycle`, + `browse`, `cast`, `download`, `crash`, `other`. +- `run` (capture_session_id) distinguishes lines from prior app runs when + breadcrumbs or debug files span sessions. +- Playback additionally emits periodic (~5s) player-stats snapshot lines + (`"cat":"playback","tag":"StatsSnapshot"`) built from the existing stats + models — a timeline of decoder, resolution, HDR mode, bitrate, dropped + frames, and audio underruns around the failure. + +### Safe logging contract (redaction happens at collection time) + +Best-effort scrubbing at package time is not enough: today's client logs +already emit full URLs with query strings, token suffixes, and raw exception +strings (e.g. the Apple `HTTPClient` response logging and player URL prints; +Android exception messages containing backend URLs). Therefore: + +- The facade accepts **typed attributes**, not preformatted strings, for + anything risky: URL values are normalized (userinfo, query, and fragment + always stripped; path preserved), exceptions pass through a sanitizing + formatter, and free-text `msg` is authored, not interpolated from network + data. +- Attribute keys are **normatively registered**: a per-category key registry + (checked in next to the JSON Schema) declares each key and its value type + (scalar types plus a few enumerated units). Unregistered keys fail debug + builds/CI and are dropped in production builds. +- Existing call sites are migrated **curated, not mechanically** — each + migrated line is reviewed against the contract. Unmigrated `Log.*`/`print` + call sites simply never enter bundles. +- Defense in depth at bundle time: values matching the account's current + access/profile tokens are replaced before packaging. +- Golden leak-fixture tests (JWTs, query tokens, signed URLs, cookies, token + fragments) run in all three repos' CI against the facade and the bundle + builder. +- The network category logs `method, templated path, status, duration`; + never headers, bodies, query strings, or full URLs. + +### Device snapshot (`device.json`) + +Fields are tri-state where a platform can't provide them: a value, +`"unknown"` (probe exists, couldn't determine), or `"not_collected"` +(no probe on this platform). Every snapshot carries `captured_at` and +`provenance` (`pre_failure` for snapshots persisted with the incident, +`post_restart` for snapshots taken at bundle-build time). + +```json +{ + "captured_at": "2026-07-19T18:22:31Z", + "provenance": "pre_failure", + "identity": { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "device":"mdarcy", "form_factor":"tv", "device_id":"…" }, + "display": { "mode":"3840x2160@59.94", "modes_supported":[…], "hdr_types":["HDR10","HLG","DV"], "wide_gamut":true }, + "audio": { "outputs":[{"type":"HDMI","encodings":["AC3","EAC3","TRUEHD"],"channels":8}], "passthrough":["AC3","EAC3","TRUEHD"], "suppressions":[…] }, + "video_codecs": [ { "mime":"video/hevc", "hw":true, "profiles":["Main10","DV P8"], "max":"4096x2160@60" } ], + "network": { "transport":"ethernet" } +} +``` + +Android sources: `AndroidDeviceMetadataProvider` (id/model/platform/app +version), `MediaCodecCapabilitiesProbe` (codec + HDR decode caps), +`AudioCapabilityManager` (aggregate passthrough). **New accessors required** +(the probes exist but don't currently export everything): current + supported +`Display.Mode` list (DisplayHdrProbe exposes HDR support only), exact +identity fields (`Build.DEVICE`, form factor), `AudioManager.getDevices()` +output enumeration, and a snapshot/export API on +`PassthroughSuppressionRegistry` (its set is currently private). + +Apple sources: `AppleDeviceIdentity.current` (persisted id, device name, +platform) and `ApplePlaybackV3Capabilities.snapshot()` (codec/HDR/device +context). **Honesty constraints:** the current `outputSnapshot()` helper is +private, reports route UIDs/types rather than full output capabilities, and +its passthrough payload is a placeholder — so Apple v1 reports audio as +route metadata with `passthrough: "unknown"` until a real probe exists, and +the private helper is refactored into an internal shared accessor so +diagnostics can reuse it without duplicating route logic. Audio route UIDs +are persistent peripheral identifiers and are one-way hashed before +inclusion. + +### Playback session correlation + +Clients include recent server playback session IDs so admins can pivot into +the existing server-side logs (`operational_logs.playback_session_id` + +the admin logs filter). Both clients currently retain only the *active* +session ID (Android `PlaybackSessionManager`, Apple `PlaybackSessionBridge`), +so each client adds a small **bounded, persisted recent-session tracker** +(last ~10 session IDs with timestamps) that survives restarts — required for +next-launch crash reports. + +## Server design (`silo-server`) + +### Feature gate + +Borrows the *behavior* of the requests gate (status endpoint + per-call +guard that rereads settings) but stores keys in the general `server_settings` +store — requests uses its own dedicated table, which diagnostics doesn't +need. Key naming follows existing namespaces (`opslog.*`, +`audiobooks.enabled`, `playback.protocol_v3_enabled`): + +- `diagnostics.uploads_enabled` (bool, default `false`) +- `diagnostics.max_bundle_bytes` (default 10 MiB compressed) +- `diagnostics.max_uncompressed_bytes` (default 64 MiB) +- `diagnostics.max_reports_per_user_per_day` (default 20) +- `diagnostics.retention_days` (default 30) +- `diagnostics.max_bytes_per_user` (default 200 MiB) + +Enabling `diagnostics.uploads_enabled` **requires a storage validation** +(private object store configured and writable); the admin settings API +refuses to enable it otherwise. `S3Private` is a nullable dependency in the +router and is only populated when configured, so the feature must degrade +predictably, not 500. + +### Status endpoint + +`GET /api/v1/diagnostics/status` — authenticated (`RequireAuth`), **no +profile required**: crashes during login/profile selection must still be +reportable, so capability discovery and upload are account-scoped with +optional profile attribution. + +```json +{ + "status": "available", // available | disabled | storage_unavailable + "server_instance_id": "srv_…", // stable; binds consent + pending reports + "accepted_schema_versions": [1], + "max_bundle_bytes": 10485760, + "max_manifest_bytes": 65536, + "retention_days": 30, + "consent_notice_version": 1 +} +``` + +Clients cache this alongside existing server-driven config refresh. + +### Ingest endpoint + +`POST /api/v1/diagnostics/reports` — `RequireAuth`; access-token +`Authorization` header only (no query credentials; `sa_` API keys rejected +for diagnostics). Optional validated profile attribution. Hardened parsing — +the avatar handler is the *precedent* (multipart + `PutObject`), but its +buffer-everything approach (`ParseMultipartForm` + `io.ReadAll`) is not +reused: + +1. Guard: `diagnostics.uploads_enabled` + storage present, else stable error + codes (`disabled`, `storage_unavailable`). +2. `http.MaxBytesReader` with `max_bundle_bytes` + manifest overhead before + any parsing; request deadline; per-account concurrency cap (1 in-flight) + and a server-global concurrency ceiling, on top of the existing rate-limit + middleware. +3. Streaming `multipart.Reader`: exactly the two named parts in order; + manifest capped at 64 KiB and schema-validated; unknown parts rejected. +4. Quota reservation: in one transaction holding a per-account advisory + lock (`pg_advisory_xact_lock(user_id)` — serializing concurrent uploads + from multiple devices on one account), check the per-day count and + per-user byte quota and insert the report row in state `receiving` + (server-generated UUID + `short_id`). Failures return `quota_exceeded` + (429, with `retry_after`) or `too_large` (413) — machine-readable so + clients back off and keep the report pending locally. +5. Stream the `bundle` part directly to its final key + (`diagnostics/{user_id}/{report_id}.tar.gz`) while hashing and counting + bytes; enforce the compressed cap during streaming. Validate the + gzip/tar structure and per-entry limits (entry count, name allowlist, + uncompressed total ≤ `max_uncompressed_bytes`, compression ratio bound) + in the same bounded streaming pass. Reject on violation + (`invalid_bundle`, `unsupported_schema`). There is no staging/rename + step — S3-style stores have no cheap rename, and the row's `state` is + the source of truth: a blob is not a report until its row is `ready`. +6. On success, update the row to `ready` with sizes + sha256. On any + failure: delete the object, mark the row `failed` (compensation); the + reconciler (below) cleans up stragglers from process death mid-request. +7. Respond `201 {"report_id":"…","short_id":"SILO-…"}` and emit a normalized + operational event (accepted/rejected, account, sizes, result). + +`short_id` is 12 Crockford-Base32 characters, server-generated with +collision retry, case-insensitively unique, shown to the user for support +conversations, filterable in the admin API — and explicitly not an +authorization secret. + +New code: `internal/diagnostics/` (service + store interface), handler in +`internal/api/handlers/diagnostics.go`, wired through `Dependencies` in +`internal/api/router.go`. The store interface is diagnostics-owned (the +avatar handler's is package-private and presign-only) and covers everything +the design uses: a **streaming** put (`io.Reader` — the existing s3client +`PutObject` takes `[]byte`, so this is a small s3client addition), +`GetObject` (streamed admin download fallback), `DeleteObject`, +`ListObjects(prefix)` (reconciler), and `PresignGetURL`. + +### Storage + +```sql +CREATE TABLE client_diagnostic_reports ( + id UUID PRIMARY KEY, + short_id TEXT NOT NULL, -- unique, case-insensitive (expression index) + user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE, + profile_id TEXT, -- capturing profile, nullable + state TEXT NOT NULL, -- receiving | ready | failed + captured_at TIMESTAMPTZ NOT NULL, -- client claim + received_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- server truth; retention keys off this + report_type TEXT NOT NULL, -- crash|anr|native_crash|hang|abnormal_exit|manual + platform TEXT NOT NULL, + app_version TEXT NOT NULL, + crash_summary TEXT, -- truncated ≤ 8 KiB + manifest JSONB NOT NULL, -- part 1 verbatim (≤ 64 KiB by construction) + playback_session_ids TEXT[] NOT NULL DEFAULT '{}', + blob_bucket TEXT, + blob_key TEXT, + blob_bytes BIGINT, + uncompressed_bytes BIGINT, + blob_sha256 TEXT +); +CREATE UNIQUE INDEX ON client_diagnostic_reports (lower(short_id)); +CREATE INDEX ON client_diagnostic_reports (user_id, received_at DESC); +CREATE INDEX ON client_diagnostic_reports (received_at); +``` + +Plain table, not partitioned — volume is orders of magnitude below +`operational_logs`; the byte-heavy payload lives in object storage. Full +stacks/traces live only in the archive; the row holds a truncated summary. + +### Admin API + +Under the acting-admin subtree (same placement as the `admin_logs.go` +routes): + +- `GET /api/v1/admin/diagnostics/reports` — list; filters: user, platform, + type, date range, `short_id` (exact, case-insensitive); paginated. +- `GET /api/v1/admin/diagnostics/reports/{id}` — row incl. manifest. +- `GET /api/v1/admin/diagnostics/reports/{id}/download` — presigned GET URL + (primary, matching avatar behavior) with a streamed `GetObject` proxy + fallback for deployments where presigned URLs aren't reachable — the + fallback is new behavior, not something the avatar path provides. +- `DELETE /api/v1/admin/diagnostics/reports/{id}` — row + blob. + +Admin download and delete emit audit/operational events. Web admin gets a +list/detail/download page next to the logs viewer, with a +playback-session-id link into the existing filtered logs view. Native apps +add no admin screens in v1 (product policy). + +### Retention + reconciliation + +A taskmanager task `internal/taskmanager/tasks/cleanup_client_diagnostics.go` +mirroring `cleanup_operational_log.go`, registered in `cmd/silo/main.go` +with the other cleanup tasks: + +- Delete reports older than `diagnostics.retention_days` (by `received_at`), + then enforce `diagnostics.max_bytes_per_user` oldest-first. Blob deleted + first (tolerating already-missing objects), then the row; deletes are + idempotent. +- Reconcile: remove `receiving`/`failed` rows older than a grace window, + and (via `ListObjects`) any `diagnostics/*` objects whose row is absent, + `failed`, or stale-`receiving` — so partial failures and process deaths + mid-request never leak storage. +- Account deletion cascades rows (FK) ; the reconciler then removes orphaned + blobs. + +## Android design (`silo-android`) + +New package +`android-shared/src/androidMain/kotlin/org/siloserver/silo/common/diagnostics/`. + +### Logging + +- `SiloLog.kt` — safe-logging facade per the contract above: + `SiloLog.d(cat, tag, msg, attrs)` (+`i/w/e`, sanitized `Throwable`). + Always forwards to `android.util.Log` (today's behavior unchanged) and + offers the entry to the ring buffer. +- `LogRing.kt` — always-on in-memory ring: fixed-size array of pre-rendered + compact entries, **non-blocking** writes (atomic index; a torn final entry + during crash snapshot is acceptable), ~4000 entries / ~1.5 MB cap, lazy + attribute formatting off the hot path, dropped-entry counter. Must add no + measurable contention to playback; a playback benchmark gate accompanies + the PR that wires player events in. +- `DiagnosticsFileLogger.kt` — only while debug logging is on: a + single-writer coroutine draining a bounded channel (drop + count under + backpressure) into **append-only framed segment files** (5 × 2 MB, + app-private `files/diagnostics/logs/`, `noBackup`), compressed only at + bundle-build time — a process death never corrupts more than the tail of + the active segment (unlike a single long-lived gzip stream). +- Playback wiring: each `PlaybackAnalyticsListener` event plus a ~5s periodic + `PlayerStatsSnapshot` line while debug logging is on (reuses the existing + reducer). TV focus logging hooks the TV app's focus observers under + `cat=focus`. Network: a Ktor hook logging `method, templated path, status, + duration_ms` only. + +### Crash and abnormal-exit capture + +- `CrashCapture.kt` — installed as early as possible in both `Application` + classes; idempotent; captures the previous + `Thread.getDefaultUncaughtExceptionHandler` once. On crash it does the + minimum on the dying thread: render the stack (bounded), copy what the + non-blocking ring exposes within a hard time budget, write **one bounded + marker file** (≤ ~512 KiB) via temp-write + atomic rename, and invoke the + prior handler in a `finally`. No locks shared with the ring's write path, + no coroutines, no allocation-heavy JSON building (preformatted line + buffer). Bundle assembly happens on next launch, never in the handler. + At most 3 pending reports are kept (oldest dropped) as a crash-loop guard. +- `ExitInfoCollector.kt` — API 30+ enhancement (minSdk is 24; many Fire TV + devices ship below API 30, where the handler is the only source): reads + `ActivityManager.getHistoricalProcessExitReasons()` on launch. + `REASON_CRASH`, `REASON_CRASH_NATIVE`, `REASON_ANR` become pending + reports. Dedupe uses a persisted bounded **fingerprint set** (process + name, pid, timestamp, reason, status, trace hash) — not a raw timestamp + watermark, which loses same-stamp events and breaks on clock changes. + `getTraceInputStream()` is treated as optional opaque bytes with a bounded + read: ANR text → `crash/stack.txt`; API 31+ native tombstone protobuf → + `crash/tombstone.pb` (never decoded into the manifest). UEH-captured + events are correlated by fingerprint so the same death isn't reported + twice. +- Next foreground launch, per the consent model: Ask → aggregated, + incident-specific prompt (one prompt covering N pending reports; default + focus **Don't send** on TV; "Always send" requires a second confirmation; + full-screen review flow on TV rather than a small dialog); Always → + silent upload, at most one bundle per crash fingerprint per day; + Never → purge. Per-fingerprint prompt suppression with backoff prevents + crash-loop prompt fatigue. Pending reports expire after 7 days — + visibly (the Diagnostics screen shows pending reports with expiry), not + silently. + +### Device snapshot + +`DeviceSnapshotCollector.kt` assembles `device.json` from the existing +probes plus the new accessors listed in the contract section (display mode +list, `Build.DEVICE`/form factor, `AudioManager.getDevices()` enumeration, +suppression-registry export). A compact snapshot is persisted with each +crash marker (`provenance: pre_failure`) since post-restart audio/display +state can differ from crash-time state. + +### Settings + consent UI + +- `DiagnosticsSettingsStore.kt` — DataStore. Deliberately **not** the + per-profile pattern of `AndroidPlayerSettingsStore`: consent and pending + reports are keyed by server+account (per the consent model), and debug + logging is per-device. Holds consent records, crash-report mode, + debug-logging flag, fingerprint set, session tracker, and status cache. +- Phone: "Diagnostics" section in `SettingsScreen.kt`; TV: rail category in + `TvSettingsScreen.kt`. For non-child profiles the section is **always + visible** and state-aware rather than hidden when the server gate is off — it distinguishes + `available` / `disabled by server` / `storage unavailable` / `offline`, + and always shows local pending reports (count, size, expiry) with + review/delete controls. Uploads are simply unavailable in non-available + states; a report captured while the gate was off is not auto-uploaded + later unless consent for that server+account exists. +- Manual capture is the primary support flow and is stronger than a bare + "send now": **Start diagnostic capture** (turns on verbose capture, + shows an indicator) → user reproduces the issue → **Stop & review** → + summary (categories, time range, sizes, destination) → send. A one-shot + "Send diagnostics now" remains for quick cases but warns when only the + in-memory ring is available ("this bundle contains only the last few + minutes of basic logs"). +- After upload, the `short_id` is displayed and kept in a small sent + history so a user can read it to their admin later. + +### Upload + +- `shared/commonMain/…/network/api/DiagnosticsApi.kt` — `getStatus()` and + `uploadReport(manifestJson, bundleBytes)` via Ktor + `MultiPartFormDataContent` (first multipart use in the client — verified + none exists — written as a small reusable helper), using the existing + `safeApiCall`/`ApiResult` pattern, wired through Koin like other `*Api` + classes. +- `DiagnosticsUploader.kt` — builds the tar.gz (marker + ring/segment files + overlapping the incident window + breadcrumbs + snapshot), streams it to + the endpoint, deletes the pending report only on 201, records `short_id`. + On `quota_exceeded`/`too_large`/`disabled`/`storage_unavailable` it keeps + the pending report and backs off per `retry_after`; on + `unsupported_schema` it surfaces "server needs an update" rather than + silently retrying until expiry. + +## Apple design (`silo-apple`) + +Mirrors Android with platform-native mechanisms, and with tvOS scoped +honestly — **MetricKit diagnostic payloads (crash/hang) are iOS-only; the +SDK marks them unavailable on tvOS**, and delivery is delayed (typically +daily, while the app runs), not "next launch". + +- **Ring + facade**: `DiagLog` implementing the same safe-logging contract. + The player's existing `cmpLog` helper (`PlayerLog.swift`) is **extended** + into a tee — it keeps `print()`ing (preserving `devicectl --console` + capture on tvOS) and also feeds the ring; remaining direct `print()` call + sites in `Screens/Player/` migrate to it curated, per the redaction + contract. OSLog-based subsystems are harvested at bundle-build time via + `OSLogStore(scope: .currentProcessIdentifier)` (deployment targets are + iOS/tvOS 26, far above the OSLogStore 15+ floor), merged into + `logs.jsonl` with provenance. +- **Breadcrumb journal**: because MetricKit reports arrive in a *later* + process, the in-memory ring is gone by delivery time. While crash + reporting is enabled, a small append-only breadcrumb journal (bounded, + rotating, crash-safe appends: lifecycle transitions, playback start/stop, + route changes, screen changes) persists across runs and becomes + `breadcrumbs.jsonl`, matched to a diagnostic by `capture_session_id` and + time window. Snapshots taken at bundle-build time are labeled + `provenance: post_restart`; the design never presents post-restart state + as crash-time state. +- **iOS crash/hang detection**: an `MXMetricManager` subscriber. Each + received `MXCrashDiagnostic`/`MXHangDiagnostic` becomes its **own** + pending report: raw diagnostic JSON preserved as `crash/metrickit.json`, + deduped by a canonical hash of the diagnostic (not timestamps), carrying + its own app version and reporting-period bounds. Prompt copy is + type-specific ("crashed" vs "was not responding") and honest about timing + ("yesterday during playback"). Delivery is verified on physical devices; + parsing is tested via injected fixtures. +- **tvOS abnormal-exit detection**: no crash stacks in v1. A dirty-exit + sentinel (marker written at launch, cleared on clean + background/termination) plus the breadcrumb journal yields + `type: abnormal_exit` reports — "Silo did not shut down cleanly last + time" — with breadcrumbs and device snapshot but no stack. An in-process + crash reporter for tvOS is explicitly deferred (see Non-goals). +- **Device snapshot**: `AppleDeviceIdentity.current` + + `ApplePlaybackV3Capabilities.snapshot()` for identity/codec/HDR context; + audio reported as route metadata with hashed route UIDs and + `passthrough: "unknown"` until a real output-capability probe exists (the + current one is a private placeholder). +- **Settings/consent**: Diagnostics section in `SettingsView` (iOS) and + `TVSettingsView` (tvOS) with the same state-aware presentation, manual + capture flow, and consent rules. +- **Upload**: a small multipart body writer added to `HTTPClient` (verified: + JSON-only today, no multipart/uploadTask), same status/error handling as + Android. +- Housekeeping note: product bundle IDs are already `org.siloserver.silo`; + only legacy keychain/logging identifiers (`com.continuum.*`) remain, so + nothing here blocks on rebranding. + +## Privacy summary + +- **Consent boundaries**: no network transmission of diagnostic data without + prompt confirmation, a server+account-bound Always election, or an + explicit manual send. The always-on ring is memory-only; persistent + capture (segments, breadcrumbs, markers) is app-private, size-capped, + backup-excluded, and purged by Never/sign-out/server removal. +- **Never collected**: credentials, `Authorization`/profile-token values, + request bodies, query strings, full URLs (normalized at collection), + unsanitized exception dumps. Bearer-token matching at bundle time as + defense in depth. Golden leak fixtures in CI. +- **Deliberately collected** (disclosed in View report): device identity + incl. device name, media titles appearing in playback logs, templated + server paths. The recipient is the user's own server admin, who already + has server-side access to this information; that is the trust model this + feature is scoped to — and why third-party destinations are out. +- **Persistent identifiers**: the existing device IDs (already sent to the + server today) plus hashed audio-route UIDs; each is enumerated in the + schema docs with its purpose and retention. +- **User control**: state-aware Diagnostics screen with pending list, + review, delete, sent history; admin-side deletion; account deletion + cascades reports; admin access is acting-admin-only and audited. +- **Bounded retention**: defaults 30 days / 200 MiB per user, admin-tunable, + enforced server-side. + +## Rollout plan (vertical slices, each independently shippable) + +1. **Server foundation** — settings + storage-validated gate, status + endpoint, hardened ingest with state machine, schema + fixtures, admin + API, retention/reconciler task, operational audit events. Feature stays + off; nothing user-visible. +2. **Contract fixtures in clients** — JSON Schema + golden fixtures wired + into Android/Apple CI; safe-logging facades + rings land **dark** (no + UI, no capture persistence, no prompts). +3. **Android end-to-end slice** — consent model + Diagnostics settings UI + + crash marker/ExitInfo capture + device snapshot + uploader + manual + capture flow, with the initial curated logging migration (player tags + + TV focus + network hook + stats snapshots). First release where a user + can be asked for, and send, a useful report. +4. **Android coverage expansion** — remaining curated call-site migration, + breadcrumbs-on-Android if wanted, prompt-fatigue tuning from the field. +5. **Apple slice** — cmpLog tee + OSLogStore harvest + breadcrumbs + + MetricKit (iOS) + exit sentinel (tvOS) + settings/consent + multipart + upload. +6. **Web admin viewer + correlation UX** — list/detail/download page, + playback-session pivot into the logs viewer. +7. **Later / separate decisions** — symbolication job, tvOS in-process + crash reporter, native-app admin viewer, macOS. + +Slices 3 and 5 ship complete consent→capture→upload flows; intermediate +releases never prompt for reports they can't send. + +## Open questions + +1. Should users be able to delete their **uploaded** reports from the client + (self-service deletion), or is pending-report control + admin deletion + enough for v1? +2. tvOS: is abnormal-exit detection (no stacks) acceptable for v1, or does + the Shield/Fire TV debugging need justify evaluating an in-process crash + reporter (PLCrashReporter/KSCrash) with its signal-safety and review + trade-offs? +3. Do we want breadcrumbs on Android in v1 (cheap once the facade exists), + or is the UEH marker + ExitInfo enough there? +4. Localization of prompt/consent strings at launch vs English-first. +5. Exact quota defaults (20/day, 200 MiB/user, 10 MiB/bundle) — tune before + enabling anywhere real. +6. Whether the status payload should also fold into an existing server-driven + config refresh response later (clients poll both today); the dedicated + endpoint ships first either way. + +## Review notes + +This spec was adversarially reviewed before the draft PR: two independent +review passes (a fact-check of every codebase claim against the three repos, +and a design attack across consent, crash-capture correctness, ingest +robustness, storage consistency, UX, and phasing). Material corrections +adopted include: tvOS MetricKit unavailability and honest MetricKit timing; +server+account-bound consent; storage-validated gating (private object store +is optional in real deployments); streaming bounded ingest with a report +state machine and orphan reconciliation; the archive container replacing a +bare log gzip; collection-time typed redaction replacing scrub-at-package; +fingerprint-based ExitInfo dedupe; non-blocking ring + bounded crash marker; +recent-session trackers (only the active playback session ID exists today); +and honest Apple snapshot capabilities (passthrough currently unknown).