haeaeae/silo-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(design): client diagnostics spec (crash reports + debug log upload)

Browse Source 
Cross-repo spec and rollout plan for opt-in client crash reporting and
debug-log upload to the user's own Silo server: silo-server ingest, storage,
admin API, and retention; silo-android and silo-apple capture, consent, and
upload. Disabled by default on both server and client.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XppCCycoaskCsW7ja1fZct
This commit is contained in:
Quick104
2026-07-19 16:27:37 -04:00
co-authored by Claude Fable 5
parent 8044eb84dd
commit 7369d0afd8
1 changed files with 766 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/design/2026-07-19-client-diagnostics.md
+766
View File
@@ -0,0 +1,766 @@
# Client Diagnostics: crash reports and debug log upload
Status: Draft (spec + cross-repo plan)
Date: 2026-07-19
Repos affected: `silo-server`, `silo-android`, `silo-apple`
## Summary
Add an opt-in, self-hosted diagnostics pipeline: Silo clients detect crashes
and abnormal exits, collect structured debug logs (playback, TV focus,
network, lifecycle) plus a hardware snapshot (device model, display/HDMI,
audio path, codec capabilities), and — only with explicit user consent —
upload a compressed diagnostics bundle to the user's own Silo server. Admins
review bundles through new acting-admin API endpoints alongside the existing
server logs tooling.
Everything ships **disabled by default**:
- Server: a `diagnostics.uploads_enabled` server setting gates the whole
feature (default off), and it cannot be enabled unless the server's private
object storage is configured and validated.
- Client: verbose "debug logging" is a per-device setting (default off).
Crash reporting defaults to **Ask** — after an abnormal exit the app shows
an incident-specific prompt (e.g. *"Silo closed unexpectedly during
playback. Send a diagnostic report to your server?"*) with
**Send / Don't send / Always send**. Nothing uploads without one of: an
explicit prompt confirmation, a prior "Always send" election bound to this
server and account, or the user explicitly sending a manual capture.
## Motivation
Today we are blind to client-side failures:
- **No crash reporting anywhere.** Neither the Android repo nor the Apple
repo installs an uncaught-exception handler, uses MetricKit, or bundles a
crash SDK (verified: no handler/SDK references in either repo; the only
Firebase dependency is messaging). A user saying "the app crashed" is a
dead end.
- **Diagnostics don't survive to the report.** Android logs go to logcat (raw
`android.util.Log` in ~40 files, no buffer or file sink); the Apple
player's rich `[CMP]`/`[CMP-DIAG]` diagnostics go through `print()`
(`PlayerLog.swift` / `PlayerCore.swift`) with no app-owned persistent sink.
By the time a user reports a problem, the evidence is gone.
- **The hardest bugs are device-specific.** Shield HLG quirks, Fire TV
passthrough suppression, HDMI mode switches, TV focus traps — reproducing
them requires knowing the exact device, panel, audio chain, and codec path.
Users can't reliably report any of that.
- **The server already has half the story.** `operational_logs` records
`playback_session_id` (migration `028_log_partitioning.sql`), the admin
logs API filters by it, and the web admin has a logs viewer. What's missing
is the client half, and a way to join the two.
## Goals
- Capture crashes and abnormal exits (JVM/native/ANR on Android; crash and
hang diagnostics on iOS; abnormal-exit detection on tvOS) with enough
context to act on: stack or exit trace where the platform provides one,
recent structured logs, and a device snapshot.
- Capture deep opt-in debug logs for playback, TV focus behavior, networking,
and navigation/lifecycle — everything needed to troubleshoot user issues.
- Always include a hardware snapshot: device (Shield / Fire TV / onn /
phone), OS, app build, display and HDMI mode with HDR capabilities, audio
output path and passthrough capabilities, decoder inventory — with honest
`unknown` / `not_collected` states where a platform can't provide a field.
- One payload contract across Android and Apple; one server endpoint; one
admin surface. The contract is a checked-in versioned JSON Schema with
golden fixtures consumed by all three repos' CI.
- Explicit, server-bound consent for every byte uploaded; feature fully off
by default on both server and client.
- Bounded cost: size caps, per-account quotas, concurrency limits, and
server-side retention.
## Non-goals (v1)
- Live/continuous log streaming from clients to the server.
- Screenshots or screen recordings in bundles.
- Native/tvOS crash *stack* capture via an in-process crash reporter
(PLCrashReporter/KSCrash-style). tvOS v1 detects abnormal exits and
attaches breadcrumbs, not stacks (see Apple design). Adopting an in-process
crash reporter is a separate decision with its own signal-safety and review
implications.
- Symbolication. Bundles store what the OS gives us; a server-side
symbolication job can come later.
- Native-app admin viewer screens (per current product policy, admin surfaces
beyond STATS on the clients are a separate product decision; v1 admin
consumption is via API + web admin).
- macOS. The Apple repo has a macOS target, but v1 scopes to
`android`, `android-tv`, `ios`, `tvos`; the platform enum reserves `macos`
without accepting it.
- Third-party telemetry, metrics aggregation, or fleet analytics. This is a
support/debugging tool, not analytics.
## Why this route
**Upload to the user's own Silo server, not a third-party crash service.**
Crashlytics/Sentry SaaS would send user data to a third party, require vendor
accounts/keys we'd have to ship, and put the reports in front of *us* rather
than the person who can actually act — the self-hosting admin. Requiring
every admin to also run self-hosted Sentry is far too heavy. The Silo server
is the natural, privacy-aligned destination, and it already has auth, admin
tooling, object storage, and retention patterns to build on.
**Bundle-on-event, not streaming.** A bundle captured at the moment of
failure (crash, or a user-triggered capture) contains exactly the context
needed: recent structured logs, the playback stats timeline, and the hardware
snapshot. Continuous streaming would cost bandwidth and privacy while mostly
capturing noise, and would still miss pre-crash buffers.
**Hybrid storage: Postgres row + object-store blob.** Report *metadata* (a
small manifest: device summary, truncated crash summary, type, timestamps)
goes in a queryable table; the payload itself (potentially megabytes of logs
and crash artifacts) goes to the S3-compatible private bucket as a single
archive. Ingesting client log lines into the partitioned `operational_logs`
table was considered and rejected: bundles are consumed as whole documents by
a human, not queried row-wise; client lines are untrusted input with
different retention needs; and multi-MB uploads would bloat a table tuned for
the server's own high-churn logs. The server *does* emit normalized
operational events about diagnostics activity (report accepted / rejected /
downloaded / deleted, with report id, account, sizes, result) so ingest and
admin actions are auditable and correlate with existing logs — raw client
lines never enter `operational_logs`.
**A versioned archive, not a bare gzipped log stream.** Crash evidence is
heterogeneous: JSONL logs, ANR trace text, Android native tombstones
(protobuf on API 31+), MetricKit diagnostic JSON. A tar.gz archive with named
entries carries each artifact losslessly with a declared media type, keeps
large traces out of Postgres, and can evolve by adding entries.
**Build small in-house rather than adopt ACRA/other SDKs.** The surface we
need on Android (one exception handler writing a bounded marker, a ring
buffer, one upload call) is a few hundred lines; ACRA would add a dependency
and its own report format without covering Apple. On Apple, the OS provides
MetricKit natively (iOS; not tvOS — see Apple design).
## Architecture overview
```
Android / Apple client silo-server
┌────────────────────────────────┐ ┌──────────────────────────────────────┐
│ Safe-logging facade → ring │ │ GET /api/v1/diagnostics/status │
│ └ debug mode: rotating │ │ └ available|disabled| │
│ segment files │ │ storage_unavailable + limits │
│ Crash/abnormal-exit capture │ │ POST /api/v1/diagnostics/reports │
│ ├ UEH marker (Android) │──▶│ ├ gate + storage check │
│ ├ ApplicationExitInfo (30+) │ │ ├ streaming, bounded multipart │
│ ├ MetricKit (iOS) │ │ ├ account quotas + concurrency │
│ └ exit sentinel (tvOS) │ │ ├ manifest → client_diagnostic_ │
│ Pending reports (server-bound) │ │ │ reports (state machine) │
│ DeviceSnapshot (probes+new │ │ └ archive → S3 private bucket │
│ accessors) │ │ Admin (acting admin): list/detail/ │
│ Consent UI (ask/always/never, │ │ download/delete + audit events │
│ per server+account) │ │ Retention + orphan reconciler task │
└────────────────────────────────┘ └──────────────────────────────────────┘
```
## Consent model
Consent is **scoped, versioned, and destination-bound** — not a global
boolean. Both clients support multiple servers and accounts, so a device-wide
"Always send" would let a bundle full of server A's titles and URLs upload to
server B after a switch. Rules:
- Consent (including "Always send") is **account-scoped**: every consent
record and every pending report is bound to
`(server_instance_id, account_user_id, consent_notice_version)`, where
`server_instance_id` is a stable opaque ID returned by the status
endpoint. The capturing `profile_id` is recorded on reports for
*attribution*, not as a consent scope — quotas and elections follow the
account.
- A pending report is only ever uploaded to the server+account it was
captured under. Reports are never retargeted. Switching server or account
neither uploads nor migrates pending reports; signing out of an account or
removing a server deletes its pending reports.
- "Always send" is an election for one server+account. Bumping
`consent_notice_version` (when we materially change what bundles contain)
invalidates prior elections and reverts to Ask.
- Setting crash reports to **Never** deletes pending reports for that
server+account and stops persistent capture (breadcrumbs/debug files); the
in-memory ring keeps running (memory-only, never uploaded without a later
explicit action).
- **Child/managed profiles**: the Diagnostics section is not shown to
profiles with `is_child`, and every diagnostics action — changing
settings, electing Always, the crash prompt, and manual capture/send —
requires a non-child profile. A crash captured while a child profile was
active is held pending and prompted for the next time a non-child profile
is active. Reports are attributed to account + capturing profile.
- The crash prompt and the manual flow both offer **View report** — a
human-readable summary (incident, categories and line counts, device
identity, destination server) before anything is sent.
- Uploads carry `consent: {mode, notice_version}` in the manifest so the
server can reject stale-consent uploads outright.
## Bundle format (shared contract)
An upload is `multipart/form-data` with exactly two parts, in order:
1. `manifest` — `application/json`, ≤ 64 KiB. Queryable summary; stored in
Postgres.
2. `bundle` — `application/gzip`: a tar.gz archive, `manifest.json` first.
Entry names come from a **fixed allowlist**, and each name implies its
media type — there is no per-entry type metadata to drift:
```
manifest.json (manifest minus the `archive` object; see hashing note)
device.json (full device snapshot)
logs.jsonl (structured log lines, newest-last)
crash/summary.json (normalized crash info incl. provenance)
crash/stack.txt (JVM stack / ANR trace, when present)
crash/tombstone.pb (Android native tombstone, opaque bytes, API 31+)
crash/metrickit.json (raw MetricKit diagnostic JSON, iOS)
breadcrumbs.jsonl (persisted pre-crash context journal, when present)
```
Hashing note: `archive.sha256` and the size fields in part 1 describe the
finalized `bundle` part bytes as transmitted. Since the archive is hashed
after it is built, the embedded `manifest.json` is the manifest **without**
the `archive` object; the server verifies part 1's `archive` fields against
the received blob.
The contract lives in this repo as a versioned JSON Schema
(`docs/design/schemas/client-diagnostics/v1/`) with canonical valid and
invalid fixtures; Go, Kotlin, and Swift CI all validate against the same
fixtures. Servers accept all schema versions they support and ignore unknown
additive fields.
### Manifest (part 1)
```json
{
"schema_version": 1,
"report": {
"type": "crash", // crash | anr | native_crash | hang | abnormal_exit | manual
"captured_at": "2026-07-19T18:22:31Z",
"capture_session_id": "run_c7d…", // UUID per app run, also stamped on log lines
"app_version": "1.4.2",
"app_build": "20841",
"platform": "android-tv", // android | android-tv | ios | tvos (macos reserved)
"os_version": "11 (API 30)",
"profile_id": "prof_…" // capturing profile, attribution only
},
"destination": { "server_instance_id": "srv_…" }, // consent binding; server rejects mismatches
"consent": { "mode": "prompt", "notice_version": 1 }, // prompt | always | manual
"crash": { // absent for manual
"summary": "NullPointerException in PlaybackSessionManager.start",
"stack_excerpt": "…truncated ≤ 8 KiB; full artifact in archive…",
"thread": "main",
"foreground": true,
"source": "ueh", // ueh | exit_info | metrickit | exit_sentinel
"provenance": "pre_failure", // pre_failure | post_restart | metric_reporting_period
"occurred_at": "2026-07-19T18:22:31Z" // best known; MetricKit reports carry period bounds
},
"device_summary": { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "os":"11", "form_factor":"tv" },
"playback_session_ids": ["ps_9f2…"],
"log_summary": { "lines": 3841, "bytes_gz": 412034, "dropped_lines": 12, "categories": ["playback","focus","network"], "debug_logging": true },
"archive": { "entries": ["manifest.json","device.json","logs.jsonl","crash/summary.json","crash/stack.txt"], "bytes": 498112, "uncompressed_bytes": 3110400, "sha256": "…" }
}
```
### Log line schema (`logs.jsonl`)
```json
{"ts":"2026-07-19T18:22:29.412Z","run":"run_c7d…","lvl":"W","cat":"playback","tag":"AudioCapabilityMgr","msg":"passthrough suppressed: HDMI sink lost TrueHD","attrs":{"sink":"HDMI","fmt":"truehd"}}
```
- `lvl`: `V|D|I|W|E`. `cat`: `playback`, `focus`, `network`, `lifecycle`,
`browse`, `cast`, `download`, `crash`, `other`.
- `run` (capture_session_id) distinguishes lines from prior app runs when
breadcrumbs or debug files span sessions.
- Playback additionally emits periodic (~5s) player-stats snapshot lines
(`"cat":"playback","tag":"StatsSnapshot"`) built from the existing stats
models — a timeline of decoder, resolution, HDR mode, bitrate, dropped
frames, and audio underruns around the failure.
### Safe logging contract (redaction happens at collection time)
Best-effort scrubbing at package time is not enough: today's client logs
already emit full URLs with query strings, token suffixes, and raw exception
strings (e.g. the Apple `HTTPClient` response logging and player URL prints;
Android exception messages containing backend URLs). Therefore:
- The facade accepts **typed attributes**, not preformatted strings, for
anything risky: URL values are normalized (userinfo, query, and fragment
always stripped; path preserved), exceptions pass through a sanitizing
formatter, and free-text `msg` is authored, not interpolated from network
data.
- Attribute keys are **normatively registered**: a per-category key registry
(checked in next to the JSON Schema) declares each key and its value type
(scalar types plus a few enumerated units). Unregistered keys fail debug
builds/CI and are dropped in production builds.
- Existing call sites are migrated **curated, not mechanically** — each
migrated line is reviewed against the contract. Unmigrated `Log.*`/`print`
call sites simply never enter bundles.
- Defense in depth at bundle time: values matching the account's current
access/profile tokens are replaced before packaging.
- Golden leak-fixture tests (JWTs, query tokens, signed URLs, cookies, token
fragments) run in all three repos' CI against the facade and the bundle
builder.
- The network category logs `method, templated path, status, duration`;
never headers, bodies, query strings, or full URLs.
### Device snapshot (`device.json`)
Fields are tri-state where a platform can't provide them: a value,
`"unknown"` (probe exists, couldn't determine), or `"not_collected"`
(no probe on this platform). Every snapshot carries `captured_at` and
`provenance` (`pre_failure` for snapshots persisted with the incident,
`post_restart` for snapshots taken at bundle-build time).
```json
{
"captured_at": "2026-07-19T18:22:31Z",
"provenance": "pre_failure",
"identity": { "manufacturer":"NVIDIA", "model":"SHIELD Android TV", "device":"mdarcy", "form_factor":"tv", "device_id":"…" },
"display": { "mode":"3840x2160@59.94", "modes_supported":[…], "hdr_types":["HDR10","HLG","DV"], "wide_gamut":true },
"audio": { "outputs":[{"type":"HDMI","encodings":["AC3","EAC3","TRUEHD"],"channels":8}], "passthrough":["AC3","EAC3","TRUEHD"], "suppressions":[…] },
"video_codecs": [ { "mime":"video/hevc", "hw":true, "profiles":["Main10","DV P8"], "max":"4096x2160@60" } ],
"network": { "transport":"ethernet" }
}
```
Android sources: `AndroidDeviceMetadataProvider` (id/model/platform/app
version), `MediaCodecCapabilitiesProbe` (codec + HDR decode caps),
`AudioCapabilityManager` (aggregate passthrough). **New accessors required**
(the probes exist but don't currently export everything): current + supported
`Display.Mode` list (DisplayHdrProbe exposes HDR support only), exact
identity fields (`Build.DEVICE`, form factor), `AudioManager.getDevices()`
output enumeration, and a snapshot/export API on
`PassthroughSuppressionRegistry` (its set is currently private).
Apple sources: `AppleDeviceIdentity.current` (persisted id, device name,
platform) and `ApplePlaybackV3Capabilities.snapshot()` (codec/HDR/device
context). **Honesty constraints:** the current `outputSnapshot()` helper is
private, reports route UIDs/types rather than full output capabilities, and
its passthrough payload is a placeholder — so Apple v1 reports audio as
route metadata with `passthrough: "unknown"` until a real probe exists, and
the private helper is refactored into an internal shared accessor so
diagnostics can reuse it without duplicating route logic. Audio route UIDs
are persistent peripheral identifiers and are one-way hashed before
inclusion.
### Playback session correlation
Clients include recent server playback session IDs so admins can pivot into
the existing server-side logs (`operational_logs.playback_session_id` +
the admin logs filter). Both clients currently retain only the *active*
session ID (Android `PlaybackSessionManager`, Apple `PlaybackSessionBridge`),
so each client adds a small **bounded, persisted recent-session tracker**
(last ~10 session IDs with timestamps) that survives restarts — required for
next-launch crash reports.
## Server design (`silo-server`)
### Feature gate
Borrows the *behavior* of the requests gate (status endpoint + per-call
guard that rereads settings) but stores keys in the general `server_settings`
store — requests uses its own dedicated table, which diagnostics doesn't
need. Key naming follows existing namespaces (`opslog.*`,
`audiobooks.enabled`, `playback.protocol_v3_enabled`):
- `diagnostics.uploads_enabled` (bool, default `false`)
- `diagnostics.max_bundle_bytes` (default 10 MiB compressed)
- `diagnostics.max_uncompressed_bytes` (default 64 MiB)
- `diagnostics.max_reports_per_user_per_day` (default 20)
- `diagnostics.retention_days` (default 30)
- `diagnostics.max_bytes_per_user` (default 200 MiB)
Enabling `diagnostics.uploads_enabled` **requires a storage validation**
(private object store configured and writable); the admin settings API
refuses to enable it otherwise. `S3Private` is a nullable dependency in the
router and is only populated when configured, so the feature must degrade
predictably, not 500.
### Status endpoint
`GET /api/v1/diagnostics/status` — authenticated (`RequireAuth`), **no
profile required**: crashes during login/profile selection must still be
reportable, so capability discovery and upload are account-scoped with
optional profile attribution.
```json
{
"status": "available", // available | disabled | storage_unavailable
"server_instance_id": "srv_…", // stable; binds consent + pending reports
"accepted_schema_versions": [1],
"max_bundle_bytes": 10485760,
"max_manifest_bytes": 65536,
"retention_days": 30,
"consent_notice_version": 1
}
```
Clients cache this alongside existing server-driven config refresh.
### Ingest endpoint
`POST /api/v1/diagnostics/reports` — `RequireAuth`; access-token
`Authorization` header only (no query credentials; `sa_` API keys rejected
for diagnostics). Optional validated profile attribution. Hardened parsing —
the avatar handler is the *precedent* (multipart + `PutObject`), but its
buffer-everything approach (`ParseMultipartForm` + `io.ReadAll`) is not
reused:
1. Guard: `diagnostics.uploads_enabled` + storage present, else stable error
codes (`disabled`, `storage_unavailable`).
2. `http.MaxBytesReader` with `max_bundle_bytes` + manifest overhead before
any parsing; request deadline; per-account concurrency cap (1 in-flight)
and a server-global concurrency ceiling, on top of the existing rate-limit
middleware.
3. Streaming `multipart.Reader`: exactly the two named parts in order;
manifest capped at 64 KiB and schema-validated; unknown parts rejected.
4. Quota reservation: in one transaction holding a per-account advisory
lock (`pg_advisory_xact_lock(user_id)` — serializing concurrent uploads
from multiple devices on one account), check the per-day count and
per-user byte quota and insert the report row in state `receiving`
(server-generated UUID + `short_id`). Failures return `quota_exceeded`
(429, with `retry_after`) or `too_large` (413) — machine-readable so
clients back off and keep the report pending locally.
5. Stream the `bundle` part directly to its final key
(`diagnostics/{user_id}/{report_id}.tar.gz`) while hashing and counting
bytes; enforce the compressed cap during streaming. Validate the
gzip/tar structure and per-entry limits (entry count, name allowlist,
uncompressed total ≤ `max_uncompressed_bytes`, compression ratio bound)
in the same bounded streaming pass. Reject on violation
(`invalid_bundle`, `unsupported_schema`). There is no staging/rename
step — S3-style stores have no cheap rename, and the row's `state` is
the source of truth: a blob is not a report until its row is `ready`.
6. On success, update the row to `ready` with sizes + sha256. On any
failure: delete the object, mark the row `failed` (compensation); the
reconciler (below) cleans up stragglers from process death mid-request.
7. Respond `201 {"report_id":"…","short_id":"SILO-…"}` and emit a normalized
operational event (accepted/rejected, account, sizes, result).
`short_id` is 12 Crockford-Base32 characters, server-generated with
collision retry, case-insensitively unique, shown to the user for support
conversations, filterable in the admin API — and explicitly not an
authorization secret.
New code: `internal/diagnostics/` (service + store interface), handler in
`internal/api/handlers/diagnostics.go`, wired through `Dependencies` in
`internal/api/router.go`. The store interface is diagnostics-owned (the
avatar handler's is package-private and presign-only) and covers everything
the design uses: a **streaming** put (`io.Reader` — the existing s3client
`PutObject` takes `[]byte`, so this is a small s3client addition),
`GetObject` (streamed admin download fallback), `DeleteObject`,
`ListObjects(prefix)` (reconciler), and `PresignGetURL`.
### Storage
```sql
CREATE TABLE client_diagnostic_reports (
id UUID PRIMARY KEY,
short_id TEXT NOT NULL, -- unique, case-insensitive (expression index)
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
profile_id TEXT, -- capturing profile, nullable
state TEXT NOT NULL, -- receiving | ready | failed
captured_at TIMESTAMPTZ NOT NULL, -- client claim
received_at TIMESTAMPTZ NOT NULL DEFAULT now(), -- server truth; retention keys off this
report_type TEXT NOT NULL, -- crash|anr|native_crash|hang|abnormal_exit|manual
platform TEXT NOT NULL,
app_version TEXT NOT NULL,
crash_summary TEXT, -- truncated ≤ 8 KiB
manifest JSONB NOT NULL, -- part 1 verbatim (≤ 64 KiB by construction)
playback_session_ids TEXT[] NOT NULL DEFAULT '{}',
blob_bucket TEXT,
blob_key TEXT,
blob_bytes BIGINT,
uncompressed_bytes BIGINT,
blob_sha256 TEXT
);
CREATE UNIQUE INDEX ON client_diagnostic_reports (lower(short_id));
CREATE INDEX ON client_diagnostic_reports (user_id, received_at DESC);
CREATE INDEX ON client_diagnostic_reports (received_at);
```
Plain table, not partitioned — volume is orders of magnitude below
`operational_logs`; the byte-heavy payload lives in object storage. Full
stacks/traces live only in the archive; the row holds a truncated summary.
### Admin API
Under the acting-admin subtree (same placement as the `admin_logs.go`
routes):
- `GET /api/v1/admin/diagnostics/reports` — list; filters: user, platform,
type, date range, `short_id` (exact, case-insensitive); paginated.
- `GET /api/v1/admin/diagnostics/reports/{id}` — row incl. manifest.
- `GET /api/v1/admin/diagnostics/reports/{id}/download` — presigned GET URL
(primary, matching avatar behavior) with a streamed `GetObject` proxy
fallback for deployments where presigned URLs aren't reachable — the
fallback is new behavior, not something the avatar path provides.
- `DELETE /api/v1/admin/diagnostics/reports/{id}` — row + blob.
Admin download and delete emit audit/operational events. Web admin gets a
list/detail/download page next to the logs viewer, with a
playback-session-id link into the existing filtered logs view. Native apps
add no admin screens in v1 (product policy).
### Retention + reconciliation
A taskmanager task `internal/taskmanager/tasks/cleanup_client_diagnostics.go`
mirroring `cleanup_operational_log.go`, registered in `cmd/silo/main.go`
with the other cleanup tasks:
- Delete reports older than `diagnostics.retention_days` (by `received_at`),
then enforce `diagnostics.max_bytes_per_user` oldest-first. Blob deleted
first (tolerating already-missing objects), then the row; deletes are
idempotent.
- Reconcile: remove `receiving`/`failed` rows older than a grace window,
and (via `ListObjects`) any `diagnostics/*` objects whose row is absent,
`failed`, or stale-`receiving` — so partial failures and process deaths
mid-request never leak storage.
- Account deletion cascades rows (FK) ; the reconciler then removes orphaned
blobs.
## Android design (`silo-android`)
New package
`android-shared/src/androidMain/kotlin/org/siloserver/silo/common/diagnostics/`.
### Logging
- `SiloLog.kt` — safe-logging facade per the contract above:
`SiloLog.d(cat, tag, msg, attrs)` (+`i/w/e`, sanitized `Throwable`).
Always forwards to `android.util.Log` (today's behavior unchanged) and
offers the entry to the ring buffer.
- `LogRing.kt` — always-on in-memory ring: fixed-size array of pre-rendered
compact entries, **non-blocking** writes (atomic index; a torn final entry
during crash snapshot is acceptable), ~4000 entries / ~1.5 MB cap, lazy
attribute formatting off the hot path, dropped-entry counter. Must add no
measurable contention to playback; a playback benchmark gate accompanies
the PR that wires player events in.
- `DiagnosticsFileLogger.kt` — only while debug logging is on: a
single-writer coroutine draining a bounded channel (drop + count under
backpressure) into **append-only framed segment files** (5 × 2 MB,
app-private `files/diagnostics/logs/`, `noBackup`), compressed only at
bundle-build time — a process death never corrupts more than the tail of
the active segment (unlike a single long-lived gzip stream).
- Playback wiring: each `PlaybackAnalyticsListener` event plus a ~5s periodic
`PlayerStatsSnapshot` line while debug logging is on (reuses the existing
reducer). TV focus logging hooks the TV app's focus observers under
`cat=focus`. Network: a Ktor hook logging `method, templated path, status,
duration_ms` only.
### Crash and abnormal-exit capture
- `CrashCapture.kt` — installed as early as possible in both `Application`
classes; idempotent; captures the previous
`Thread.getDefaultUncaughtExceptionHandler` once. On crash it does the
minimum on the dying thread: render the stack (bounded), copy what the
non-blocking ring exposes within a hard time budget, write **one bounded
marker file** (≤ ~512 KiB) via temp-write + atomic rename, and invoke the
prior handler in a `finally`. No locks shared with the ring's write path,
no coroutines, no allocation-heavy JSON building (preformatted line
buffer). Bundle assembly happens on next launch, never in the handler.
At most 3 pending reports are kept (oldest dropped) as a crash-loop guard.
- `ExitInfoCollector.kt` — API 30+ enhancement (minSdk is 24; many Fire TV
devices ship below API 30, where the handler is the only source): reads
`ActivityManager.getHistoricalProcessExitReasons()` on launch.
`REASON_CRASH`, `REASON_CRASH_NATIVE`, `REASON_ANR` become pending
reports. Dedupe uses a persisted bounded **fingerprint set** (process
name, pid, timestamp, reason, status, trace hash) — not a raw timestamp
watermark, which loses same-stamp events and breaks on clock changes.
`getTraceInputStream()` is treated as optional opaque bytes with a bounded
read: ANR text → `crash/stack.txt`; API 31+ native tombstone protobuf →
`crash/tombstone.pb` (never decoded into the manifest). UEH-captured
events are correlated by fingerprint so the same death isn't reported
twice.
- Next foreground launch, per the consent model: Ask → aggregated,
incident-specific prompt (one prompt covering N pending reports; default
focus **Don't send** on TV; "Always send" requires a second confirmation;
full-screen review flow on TV rather than a small dialog); Always →
silent upload, at most one bundle per crash fingerprint per day;
Never → purge. Per-fingerprint prompt suppression with backoff prevents
crash-loop prompt fatigue. Pending reports expire after 7 days —
visibly (the Diagnostics screen shows pending reports with expiry), not
silently.
### Device snapshot
`DeviceSnapshotCollector.kt` assembles `device.json` from the existing
probes plus the new accessors listed in the contract section (display mode
list, `Build.DEVICE`/form factor, `AudioManager.getDevices()` enumeration,
suppression-registry export). A compact snapshot is persisted with each
crash marker (`provenance: pre_failure`) since post-restart audio/display
state can differ from crash-time state.
### Settings + consent UI
- `DiagnosticsSettingsStore.kt` — DataStore. Deliberately **not** the
per-profile pattern of `AndroidPlayerSettingsStore`: consent and pending
reports are keyed by server+account (per the consent model), and debug
logging is per-device. Holds consent records, crash-report mode,
debug-logging flag, fingerprint set, session tracker, and status cache.
- Phone: "Diagnostics" section in `SettingsScreen.kt`; TV: rail category in
`TvSettingsScreen.kt`. For non-child profiles the section is **always
visible** and state-aware rather than hidden when the server gate is off — it distinguishes
`available` / `disabled by server` / `storage unavailable` / `offline`,
and always shows local pending reports (count, size, expiry) with
review/delete controls. Uploads are simply unavailable in non-available
states; a report captured while the gate was off is not auto-uploaded
later unless consent for that server+account exists.
- Manual capture is the primary support flow and is stronger than a bare
"send now": **Start diagnostic capture** (turns on verbose capture,
shows an indicator) → user reproduces the issue → **Stop & review** →
summary (categories, time range, sizes, destination) → send. A one-shot
"Send diagnostics now" remains for quick cases but warns when only the
in-memory ring is available ("this bundle contains only the last few
minutes of basic logs").
- After upload, the `short_id` is displayed and kept in a small sent
history so a user can read it to their admin later.
### Upload
- `shared/commonMain/…/network/api/DiagnosticsApi.kt` — `getStatus()` and
`uploadReport(manifestJson, bundleBytes)` via Ktor
`MultiPartFormDataContent` (first multipart use in the client — verified
none exists — written as a small reusable helper), using the existing
`safeApiCall`/`ApiResult` pattern, wired through Koin like other `*Api`
classes.
- `DiagnosticsUploader.kt` — builds the tar.gz (marker + ring/segment files
overlapping the incident window + breadcrumbs + snapshot), streams it to
the endpoint, deletes the pending report only on 201, records `short_id`.
On `quota_exceeded`/`too_large`/`disabled`/`storage_unavailable` it keeps
the pending report and backs off per `retry_after`; on
`unsupported_schema` it surfaces "server needs an update" rather than
silently retrying until expiry.
## Apple design (`silo-apple`)
Mirrors Android with platform-native mechanisms, and with tvOS scoped
honestly — **MetricKit diagnostic payloads (crash/hang) are iOS-only; the
SDK marks them unavailable on tvOS**, and delivery is delayed (typically
daily, while the app runs), not "next launch".
- **Ring + facade**: `DiagLog` implementing the same safe-logging contract.
The player's existing `cmpLog` helper (`PlayerLog.swift`) is **extended**
into a tee — it keeps `print()`ing (preserving `devicectl --console`
capture on tvOS) and also feeds the ring; remaining direct `print()` call
sites in `Screens/Player/` migrate to it curated, per the redaction
contract. OSLog-based subsystems are harvested at bundle-build time via
`OSLogStore(scope: .currentProcessIdentifier)` (deployment targets are
iOS/tvOS 26, far above the OSLogStore 15+ floor), merged into
`logs.jsonl` with provenance.
- **Breadcrumb journal**: because MetricKit reports arrive in a *later*
process, the in-memory ring is gone by delivery time. While crash
reporting is enabled, a small append-only breadcrumb journal (bounded,
rotating, crash-safe appends: lifecycle transitions, playback start/stop,
route changes, screen changes) persists across runs and becomes
`breadcrumbs.jsonl`, matched to a diagnostic by `capture_session_id` and
time window. Snapshots taken at bundle-build time are labeled
`provenance: post_restart`; the design never presents post-restart state
as crash-time state.
- **iOS crash/hang detection**: an `MXMetricManager` subscriber. Each
received `MXCrashDiagnostic`/`MXHangDiagnostic` becomes its **own**
pending report: raw diagnostic JSON preserved as `crash/metrickit.json`,
deduped by a canonical hash of the diagnostic (not timestamps), carrying
its own app version and reporting-period bounds. Prompt copy is
type-specific ("crashed" vs "was not responding") and honest about timing
("yesterday during playback"). Delivery is verified on physical devices;
parsing is tested via injected fixtures.
- **tvOS abnormal-exit detection**: no crash stacks in v1. A dirty-exit
sentinel (marker written at launch, cleared on clean
background/termination) plus the breadcrumb journal yields
`type: abnormal_exit` reports — "Silo did not shut down cleanly last
time" — with breadcrumbs and device snapshot but no stack. An in-process
crash reporter for tvOS is explicitly deferred (see Non-goals).
- **Device snapshot**: `AppleDeviceIdentity.current` +
`ApplePlaybackV3Capabilities.snapshot()` for identity/codec/HDR context;
audio reported as route metadata with hashed route UIDs and
`passthrough: "unknown"` until a real output-capability probe exists (the
current one is a private placeholder).
- **Settings/consent**: Diagnostics section in `SettingsView` (iOS) and
`TVSettingsView` (tvOS) with the same state-aware presentation, manual
capture flow, and consent rules.
- **Upload**: a small multipart body writer added to `HTTPClient` (verified:
JSON-only today, no multipart/uploadTask), same status/error handling as
Android.
- Housekeeping note: product bundle IDs are already `org.siloserver.silo`;
only legacy keychain/logging identifiers (`com.continuum.*`) remain, so
nothing here blocks on rebranding.
## Privacy summary
- **Consent boundaries**: no network transmission of diagnostic data without
prompt confirmation, a server+account-bound Always election, or an
explicit manual send. The always-on ring is memory-only; persistent
capture (segments, breadcrumbs, markers) is app-private, size-capped,
backup-excluded, and purged by Never/sign-out/server removal.
- **Never collected**: credentials, `Authorization`/profile-token values,
request bodies, query strings, full URLs (normalized at collection),
unsanitized exception dumps. Bearer-token matching at bundle time as
defense in depth. Golden leak fixtures in CI.
- **Deliberately collected** (disclosed in View report): device identity
incl. device name, media titles appearing in playback logs, templated
server paths. The recipient is the user's own server admin, who already
has server-side access to this information; that is the trust model this
feature is scoped to — and why third-party destinations are out.
- **Persistent identifiers**: the existing device IDs (already sent to the
server today) plus hashed audio-route UIDs; each is enumerated in the
schema docs with its purpose and retention.
- **User control**: state-aware Diagnostics screen with pending list,
review, delete, sent history; admin-side deletion; account deletion
cascades reports; admin access is acting-admin-only and audited.
- **Bounded retention**: defaults 30 days / 200 MiB per user, admin-tunable,
enforced server-side.
## Rollout plan (vertical slices, each independently shippable)
1. **Server foundation** — settings + storage-validated gate, status
endpoint, hardened ingest with state machine, schema + fixtures, admin
API, retention/reconciler task, operational audit events. Feature stays
off; nothing user-visible.
2. **Contract fixtures in clients** — JSON Schema + golden fixtures wired
into Android/Apple CI; safe-logging facades + rings land **dark** (no
UI, no capture persistence, no prompts).
3. **Android end-to-end slice** — consent model + Diagnostics settings UI +
crash marker/ExitInfo capture + device snapshot + uploader + manual
capture flow, with the initial curated logging migration (player tags +
TV focus + network hook + stats snapshots). First release where a user
can be asked for, and send, a useful report.
4. **Android coverage expansion** — remaining curated call-site migration,
breadcrumbs-on-Android if wanted, prompt-fatigue tuning from the field.
5. **Apple slice** — cmpLog tee + OSLogStore harvest + breadcrumbs +
MetricKit (iOS) + exit sentinel (tvOS) + settings/consent + multipart
upload.
6. **Web admin viewer + correlation UX** — list/detail/download page,
playback-session pivot into the logs viewer.
7. **Later / separate decisions** — symbolication job, tvOS in-process
crash reporter, native-app admin viewer, macOS.
Slices 3 and 5 ship complete consent→capture→upload flows; intermediate
releases never prompt for reports they can't send.
## Open questions
1. Should users be able to delete their **uploaded** reports from the client
(self-service deletion), or is pending-report control + admin deletion
enough for v1?
2. tvOS: is abnormal-exit detection (no stacks) acceptable for v1, or does
the Shield/Fire TV debugging need justify evaluating an in-process crash
reporter (PLCrashReporter/KSCrash) with its signal-safety and review
trade-offs?
3. Do we want breadcrumbs on Android in v1 (cheap once the facade exists),
or is the UEH marker + ExitInfo enough there?
4. Localization of prompt/consent strings at launch vs English-first.
5. Exact quota defaults (20/day, 200 MiB/user, 10 MiB/bundle) — tune before
enabling anywhere real.
6. Whether the status payload should also fold into an existing server-driven
config refresh response later (clients poll both today); the dedicated
endpoint ships first either way.
## Review notes
This spec was adversarially reviewed before the draft PR: two independent
review passes (a fact-check of every codebase claim against the three repos,
and a design attack across consent, crash-capture correctness, ingest
robustness, storage consistency, UX, and phasing). Material corrections
adopted include: tvOS MetricKit unavailability and honest MetricKit timing;
server+account-bound consent; storage-validated gating (private object store
is optional in real deployments); streaming bounded ingest with a report
state machine and orphan reconciliation; the archive container replacing a
bare log gzip; collection-time typed redaction replacing scrub-at-package;
fingerprint-based ExitInfo dedupe; non-blocking ring + bounded crash marker;
recent-session trackers (only the active playback session ID exists today);
and honest Apple snapshot capabilities (passthrough currently unknown).