silo-server/docs/architecture/observability.md

Observability (OpenTelemetry logs + traces)

Silo emits structured logs and distributed traces via OpenTelemetry (OTLP), in addition to the existing stderr and opslog database pipeline. The feature is opt-in and default-off: with no OTEL_* / SILO_OTEL_ENABLED configuration the server behaves exactly as before (stderr + opslog only).

Metrics are not part of OpenTelemetry here. They remain on Prometheus (client_golang, the /metrics endpoint, and the domain metrics in internal/api/middleware). See Metrics below.

Enabling it

Telemetry turns on when either SILO_OTEL_ENABLED is truthy or OTEL_EXPORTER_OTLP_ENDPOINT is set.

Variable	Purpose	Default
SILO_OTEL_ENABLED	Master gate (1/true/yes/on).	off
OTEL_EXPORTER_OTLP_ENDPOINT	Collector endpoint; also implicitly enables telemetry.	—
OTEL_EXPORTER_OTLP_PROTOCOL	grpc (default) or http/protobuf.	grpc
OTEL_SERVICE_NAME	service.name resource attribute.	silo-server
OTEL_SERVICE_VERSION	service.version resource attribute.	unset
OTEL_TRACES_SAMPLER	always_on, always_off, traceidratio, parentbased_always_on, parentbased_always_off, or parentbased_traceidratio. Unsupported values (e.g. jaeger_remote) fall back to the default.	parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG	Trace-id ratio for the ratio-based samplers (0–1; clamps >1 to 1).	1.0

The node identity is attached as the service.instance.id resource attribute, so multiple Silo nodes sharing one service.name stay distinguishable in the backend.

All other OTEL_EXPORTER_OTLP_* knobs (headers, TLS, per-signal endpoints) are read directly from the environment by the OTLP exporters — the environment is the single source of truth for exporter wiring.

The endpoint/exporter connection is lazy and non-blocking: an unreachable collector does not delay or crash startup. Setup failure is also non-fatal: if the OTEL_* environment is malformed (e.g. an unparseable endpoint URL), the server logs an error and keeps running with telemetry disabled rather than crash-looping.

Architecture

The bootstrap lives in internal/telemetry (Setup in telemetry.go). When enabled it builds one shared resource.Resource, a TracerProvider (sampler per OTEL_TRACES_SAMPLER, batched OTLP exporter), a LoggerProvider (batched OTLP exporter), and the W3C TraceContext + Baggage propagator. Shutdown is deferred in cmd/silo/main.go with a flush timeout so buffered spans/logs drain on exit.

Log handler chain

Logs are bridged, not rerouted. The OTel otelslog handler is added to the existing slog handler chain via fan-out, so every record still reaches stderr and the opslog DB/admin-stream pipeline unchanged:

slog.<Level>Context(ctx, …)
  → opslog.Handler            (DB capture + admin stream, when level ≥ capture)
    → logfilter.Handler       (drops quieted subsystem prefixes)
      → slog.MultiHandler
        ├── stderr (json/text)
        └── otelslog → OTLP    (level-gated + best-effort)

Three properties matter:

• Level-gated. slog.MultiHandler.Enabled ORs its children, so the OTel branch is wrapped in a level gate bound to the shared LevelVar; console and OTLP share one verbosity knob. Without this, Debug records would be built and exported even at log_level=info.
• Best-effort. The OTel branch never propagates an export error, so a failing collector cannot break the console or DB branches.
• Redacted. The whole fan-out is wrapped in internal/logredact, so console and OTLP emit secret-masked output. Redaction is key-based (password, secret, token, api_key, authorization, cookie, …); a matching attribute's value becomes [REDACTED], including attrs bound via .With(...) and nested groups. The marker list is shared with the opslog DB path (opslog.shouldRedact → logredact.SecretKey) so all sinks agree. Limitation: values are not scanned, so a secret embedded in a free-text message or under a non-secret key is not caught.

Logging conventions (enforced)

Use the context-carrying slog variants and tag the subsystem:

slog.InfoContext(ctx, "scanner: starting", "component", "scanner", "folder_id", id)

Rules, enforced by sloglint in make lint (.golangci.yml):

• slog.<Level>Context(ctx, …) wherever a context.Context is in scope (so records carry the active trace_id/span_id). The plain slog.Info(…) form is only allowed where no ctx exists (early boot, top-level goroutines).
• Static message — the message is a constant; move dynamic parts to attributes.
• snake_case attribute keys — component, request_id, trace_id, user_id, …
• No mixed args — don't combine key-value pairs and slog.Attr in one call.

Component classification

Every direct slog.*Context call carries a component attribute. opslog classifies records by that attribute first, falling back to opslog.InferComponent (the subsystem: prefix in the message) when no attribute is present — bound loggers that already set component via .With(...) are left as-is.

Limitation: sloglint enforces the shape (context variant, snake keys, static message) but cannot enforce that a component attribute is present. That convention rests on this doc, the canonical list below, and review.

Canonical component values (first path segment under internal/; app for cmd/silo):

activitylog, adminjob, ai, api, app, audiobooks, auth, autoscan, catalog, chapterthumbs, downloads, ebooks, historyimport, jellycompat, libraryingest, manga, metadata, nodeconfig, nodepool, noderecipe, nodesessions, notifications, opslog, playback, plugins, policy, proxy, ratelimit, recommendations, requests, scanner, scanqueue, sections, taskmanager, telemetry, transcodenode, watchlist, watchsync, webhooksync, worker.

Metrics stay on Prometheus

OpenTelemetry here installs no MeterProvider. Metrics continue to flow through the existing Prometheus client_golang instrumentation to /metrics; Grafana scraping is unaffected.

This is deliberate and guarded: the trace-instrumentation libraries (otelhttp, etc., added in later phases) also emit metrics through the global MeterProvider. Because none is set, that global stays the built-in no-op, so those metric calls are silently discarded — no double-counting into Prometheus, no second /metrics source. A guard test asserts otel.GetMeterProvider() remains the no-op after Setup. Migrating metrics to the OTel metrics SDK is out of scope.

Local collector (example)

# docker-compose.yml (dev)
otel-collector:
  image: otel/opentelemetry-collector:latest
  command: ["--config=/etc/otelcol/config.yaml"]
  volumes:
    - ./otelcol.yaml:/etc/otelcol/config.yaml
  ports:
    - "4317:4317"   # OTLP gRPC
    - "4318:4318"   # OTLP HTTP

# otelcol.yaml — minimal debug pipeline
receivers:
  otlp:
    protocols:
      grpc: {}
      http: {}
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [debug] }
    logs:   { receivers: [otlp], exporters: [debug] }

Run Silo with SILO_OTEL_ENABLED=1 OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 and watch logs + traces arrive at the collector. Swap debug for Loki/Tempo (or a vendor OTLP endpoint) for real storage; Grafana can then show traces alongside the unchanged Prometheus metrics.

Log retention & rotation

Silo does not write or rotate its own log files. Rotation/retention is handled by the layer that owns each sink, which is the intended cloud-native split:

• Console (stderr) is owned by the container runtime. Under Docker, configure the json-file driver to cap and rotate on-disk logs, and mount the log directory on a volume so history survives container recreation (the problem that motivated this work):

  # docker-compose.yml
  services:
    silo:
      logging:
        driver: json-file
        options:
          max-size: "50m"   # rotate at 50 MB
          max-file: "5"     # keep 5 rotated files
  

  For a non-Docker deployment, run under a supervisor/journald or pipe to logrotate.

• OTLP is the durable, queryable path: the collector/backend (Loki, a vendor, …) owns retention and is the recommended place to keep searchable history. Enable it (see above) and set retention on the backend.

• opslog (Postgres) self-prunes: daily partitions with per-component retention and size caps (internal/opslog/cleanup.go). No operator action needed.

An in-process rotating file sink (e.g. lumberjack) was deliberately not added — it would re-introduce a custom sink the OTLP + runtime split already covers.

Known limitations

• log_quiet also suppresses OTel logs. The fan-out sits below the quiet filter, so quieting a subsystem empties its OTLP stream too (OTLP mirrors the console stream).
• Redaction is key-based, not value-based. Secrets under a recognized key are masked on every sink, but a secret embedded in a message string or under an unrecognized key is not caught (see Redacted).
• Early-boot logs are not exported. Records emitted before the handler is installed (DB connect, migrations, tuning) reach stderr only, matching existing opslog behavior.
• Per-subsystem trace propagation into plugins is a follow-up owned by silo-plugin-sdk; this repo instruments only the host side.