Files
silo-server/internal/telemetry/config.go
203a18ae83 feat(observability): OpenTelemetry logs+traces with secret redaction and slog standardization (#290)
* feat(observability): OpenTelemetry logs+traces with secret redaction

Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing
stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with
no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged.

Bootstrap (internal/telemetry):
- Setup() builds one shared resource, a TracerProvider (parent-based trace-id
  ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator
  from env. It installs NO MeterProvider — metrics stay on Prometheus, and the
  built-in no-op global MeterProvider keeps the trace instrumentation libs from
  double-emitting. Shutdown is deferred with a flush timeout.
- Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the
  shared LevelVar and best-effort so a failing collector can't break the console
  or DB branches. stderr + opslog stay untouched.

Secret redaction (internal/logredact):
- A slog.Handler masks secret-keyed attributes (password, token, api_key,
  authorization, cookie, ...) — including .With-bound attrs, nested groups,
  secret-keyed group subtrees, and values behind a LogValuer — on the console
  and OTLP sinks, with a no-op fast path when a record has no secret keys.
  opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one
  marker list.

Rotation is infra-managed (no custom file sink): container runtime for stderr,
collector/backend for OTLP, opslog partition-pruning for the DB. Documented in
docs/architecture/observability.md.

Verification: go build ./..., go vet, gofmt -l — clean; go test
./internal/telemetry/ ./internal/logredact/ -race pass.

AI-use disclosure: implemented with AI assistance (Claude Code), including
adversarial reviews that hardened the bootstrap and fixed two redaction leak
paths; reviewed by the author.

* refactor(observability): slog context+component sweep, sloglint gate (phase 3)

Part of #265. Builds on the OTel bootstrap + redaction commit.

Standardizes every log call site onto the context-carrying slog variants so
records correlate with the active OpenTelemetry trace, and locks the standard
in with a machine gate so future code (human- or AI-authored) can't drift back.

- Call-site sweep: converted the remaining slog.<Level>(...) calls to the
  slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope
  (background/init calls with no ctx are left as-is), across 183 files. Applied
  via a type-aware AST codemod. Log levels and message strings are preserved
  verbatim; a component attr (canonical per-package name) is added to direct
  package-level slog calls. Bound-logger calls keep their existing .With
  bindings. The main.go and telemetry package conversions rode with their file
  in the previous commit to keep each file within a single commit.
- Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg,
  key-naming-case=snake, no-mixed-args. After the sweep all four report zero
  violations repo-wide (tests included), so make lint / CI now blocks any
  regression to the non-context form. The gate ships with the sweep because it
  cannot be green until the legacy sites are converted.

Metrics remain on Prometheus; no behavior change to /metrics or Grafana.

Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4
rules) 0 violations repo-wide; log levels verified unchanged.

AI-use disclosure: implemented with AI assistance (Claude Code), including the
codemod; reviewed by the author.

* fix(observability): honor per-signal OTLP protocol and secret WithGroup names

Two Codex review findings on PR #290:

- telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the
  generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector
  setups (e.g. HTTP logs + gRPC traces) build the right exporter.
- logredact: entering a group whose name is secret-bearing (e.g.
  WithGroup("authorization")) now masks every leaf in that subtree,
  matching how slog.Group("authorization", ...) is masked as a whole.

* fix(observability): address review feedback on telemetry bootstrap

- Telemetry setup failure no longer kills boot: Setup returns usable
  no-op providers alongside the error and main logs and continues with
  telemetry disabled, honoring the best-effort contract.
- Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_*
  variants); unsupported values fall back to parentbased_traceidratio.
- Attach node identity as semconv service.instance.id instead of the
  non-semconv node.name.
- Rename opslog retention-scope log attrs to target_component/target_level
  so they no longer collide with the canonical component routing key, and
  tag those lines with component=opslog.
- Fix stale levelGated comment casing; use WarnContext in the telemetry
  shutdown defer; document the LogValuer double-resolve on the redaction
  slow path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 08:53:52 -04:00

169 lines
6.4 KiB
Go

// Package telemetry provides the OpenTelemetry SDK bootstrap for Silo: shared
// resource, tracer provider, logger provider, and W3C propagation. It
// deliberately does NOT build or register a MeterProvider — metrics stay on the
// existing Prometheus rail. Leaving the global MeterProvider as the built-in
// no-op is what prevents the trace instrumentation libraries from double-emitting
// metrics. See docs/superpowers/plans/2026-07-02-opentelemetry-observability.md.
package telemetry
import (
"math"
"os"
"strconv"
"strings"
)
// Protocol identifies the OTLP exporter wire protocol.
type Protocol string
const (
// ProtocolGRPC is the OTLP/gRPC protocol (default).
ProtocolGRPC Protocol = "grpc"
// ProtocolHTTP is the OTLP/HTTP+protobuf protocol.
ProtocolHTTP Protocol = "http/protobuf"
)
// Sampler identifies the head-sampling strategy (OTEL_TRACES_SAMPLER).
type Sampler string
const (
// SamplerAlwaysOn samples every trace.
SamplerAlwaysOn Sampler = "always_on"
// SamplerAlwaysOff samples nothing.
SamplerAlwaysOff Sampler = "always_off"
// SamplerTraceIDRatio samples by trace-id ratio regardless of the parent.
SamplerTraceIDRatio Sampler = "traceidratio"
// SamplerParentBasedAlwaysOn honors the parent decision, sampling roots.
SamplerParentBasedAlwaysOn Sampler = "parentbased_always_on"
// SamplerParentBasedAlwaysOff honors the parent decision, dropping roots.
SamplerParentBasedAlwaysOff Sampler = "parentbased_always_off"
// SamplerParentBasedTraceIDRatio honors the parent decision, sampling roots
// by trace-id ratio. This is the default.
SamplerParentBasedTraceIDRatio Sampler = "parentbased_traceidratio"
)
// defaultServiceName is used when OTEL_SERVICE_NAME is unset.
const defaultServiceName = "silo-server"
// defaultSamplerRatio is the parent-based trace-id ratio applied when
// OTEL_TRACES_SAMPLER_ARG is unset or unparseable.
const defaultSamplerRatio = 1.0
// Config is the fully-defaulted telemetry configuration parsed from the
// environment. It is cheap to construct and safe to build even when telemetry
// is disabled.
type Config struct {
// Enabled gates the entire feature. True when SILO_OTEL_ENABLED is truthy
// OR OTEL_EXPORTER_OTLP_ENDPOINT is set.
Enabled bool
// Endpoint is the OTLP collector endpoint (OTEL_EXPORTER_OTLP_ENDPOINT). It
// is used ONLY to decide Enabled; the exporters themselves read the endpoint
// (and all other OTEL_EXPORTER_OTLP_* knobs) directly from the environment,
// which remains the single source of truth for exporter wiring.
Endpoint string
// Protocol is the generic OTLP wire protocol (OTEL_EXPORTER_OTLP_PROTOCOL).
// It is the fallback for any signal without a signal-specific override.
Protocol Protocol
// TracesProtocol selects the trace exporter's wire protocol, honoring
// OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and falling back to Protocol.
TracesProtocol Protocol
// LogsProtocol selects the log exporter's wire protocol, honoring
// OTEL_EXPORTER_OTLP_LOGS_PROTOCOL and falling back to Protocol.
LogsProtocol Protocol
// ServiceName populates the service.name resource attribute.
ServiceName string
// ServiceVersion populates the service.version resource attribute.
ServiceVersion string
// NodeID populates the service.instance.id resource attribute.
NodeID string
// Sampler is the head-sampling strategy (OTEL_TRACES_SAMPLER). Unrecognized
// or unsupported values (e.g. jaeger_remote) fall back to
// parentbased_traceidratio.
Sampler Sampler
// SamplerRatio is the trace-id-ratio sampling probability used by the
// ratio-based samplers (OTEL_TRACES_SAMPLER_ARG).
SamplerRatio float64
}
// LoadConfig parses the telemetry configuration from the environment. nodeID is
// the resolved node identity used for the service.instance.id resource
// attribute.
func LoadConfig(nodeID string) Config {
endpoint := strings.TrimSpace(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"))
enabled := truthy(os.Getenv("SILO_OTEL_ENABLED")) || endpoint != ""
serviceName := strings.TrimSpace(os.Getenv("OTEL_SERVICE_NAME"))
if serviceName == "" {
serviceName = defaultServiceName
}
// The generic protocol is the fallback; per-signal env vars override it for
// their own exporter so mixed collector setups (e.g. HTTP logs, gRPC traces)
// work as the OTLP spec prescribes.
protocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_PROTOCOL"), ProtocolGRPC)
tracesProtocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"), protocol)
logsProtocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_LOGS_PROTOCOL"), protocol)
ratio := defaultSamplerRatio
if raw := strings.TrimSpace(os.Getenv("OTEL_TRACES_SAMPLER_ARG")); raw != "" {
// Accept only finite, non-negative values; clamp above 1 to 1.0 so a
// typo'd or +Inf arg means "sample everything" rather than silently
// falling through. NaN and -Inf fail the v >= 0 / IsInf checks.
if v, err := strconv.ParseFloat(raw, 64); err == nil && v >= 0 && !math.IsInf(v, 1) {
ratio = math.Min(v, 1.0)
}
}
return Config{
Enabled: enabled,
Endpoint: endpoint,
Protocol: protocol,
TracesProtocol: tracesProtocol,
LogsProtocol: logsProtocol,
ServiceName: serviceName,
ServiceVersion: strings.TrimSpace(os.Getenv("OTEL_SERVICE_VERSION")),
NodeID: nodeID,
Sampler: parseSampler(os.Getenv("OTEL_TRACES_SAMPLER")),
SamplerRatio: ratio,
}
}
// parseSampler maps an OTEL_TRACES_SAMPLER value to a Sampler, falling back to
// parentbased_traceidratio when the value is empty, unrecognized, or names a
// sampler this bootstrap does not support (e.g. jaeger_remote).
func parseSampler(raw string) Sampler {
switch s := Sampler(strings.ToLower(strings.TrimSpace(raw))); s {
case SamplerAlwaysOn, SamplerAlwaysOff, SamplerTraceIDRatio,
SamplerParentBasedAlwaysOn, SamplerParentBasedAlwaysOff:
return s
default:
return SamplerParentBasedTraceIDRatio
}
}
// parseProtocol maps an OTEL_EXPORTER_OTLP*_PROTOCOL value to a Protocol,
// returning fallback when the value is empty or unrecognized.
func parseProtocol(raw string, fallback Protocol) Protocol {
switch strings.ToLower(strings.TrimSpace(raw)) {
case "http/protobuf", "http":
return ProtocolHTTP
case "grpc":
return ProtocolGRPC
default:
return fallback
}
}
// truthy reports whether an env value should be treated as a boolean true.
func truthy(v string) bool {
switch strings.ToLower(strings.TrimSpace(v)) {
case "1", "true", "yes", "on":
return true
default:
return false
}
}