* feat(observability): OpenTelemetry logs+traces with secret redaction Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged. Bootstrap (internal/telemetry): - Setup() builds one shared resource, a TracerProvider (parent-based trace-id ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator from env. It installs NO MeterProvider — metrics stay on Prometheus, and the built-in no-op global MeterProvider keeps the trace instrumentation libs from double-emitting. Shutdown is deferred with a flush timeout. - Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the shared LevelVar and best-effort so a failing collector can't break the console or DB branches. stderr + opslog stay untouched. Secret redaction (internal/logredact): - A slog.Handler masks secret-keyed attributes (password, token, api_key, authorization, cookie, ...) — including .With-bound attrs, nested groups, secret-keyed group subtrees, and values behind a LogValuer — on the console and OTLP sinks, with a no-op fast path when a record has no secret keys. opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one marker list. Rotation is infra-managed (no custom file sink): container runtime for stderr, collector/backend for OTLP, opslog partition-pruning for the DB. Documented in docs/architecture/observability.md. Verification: go build ./..., go vet, gofmt -l — clean; go test ./internal/telemetry/ ./internal/logredact/ -race pass. AI-use disclosure: implemented with AI assistance (Claude Code), including adversarial reviews that hardened the bootstrap and fixed two redaction leak paths; reviewed by the author. * refactor(observability): slog context+component sweep, sloglint gate (phase 3) Part of #265. Builds on the OTel bootstrap + redaction commit. Standardizes every log call site onto the context-carrying slog variants so records correlate with the active OpenTelemetry trace, and locks the standard in with a machine gate so future code (human- or AI-authored) can't drift back. - Call-site sweep: converted the remaining slog.<Level>(...) calls to the slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope (background/init calls with no ctx are left as-is), across 183 files. Applied via a type-aware AST codemod. Log levels and message strings are preserved verbatim; a component attr (canonical per-package name) is added to direct package-level slog calls. Bound-logger calls keep their existing .With bindings. The main.go and telemetry package conversions rode with their file in the previous commit to keep each file within a single commit. - Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg, key-naming-case=snake, no-mixed-args. After the sweep all four report zero violations repo-wide (tests included), so make lint / CI now blocks any regression to the non-context form. The gate ships with the sweep because it cannot be green until the legacy sites are converted. Metrics remain on Prometheus; no behavior change to /metrics or Grafana. Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4 rules) 0 violations repo-wide; log levels verified unchanged. AI-use disclosure: implemented with AI assistance (Claude Code), including the codemod; reviewed by the author. * fix(observability): honor per-signal OTLP protocol and secret WithGroup names Two Codex review findings on PR #290: - telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector setups (e.g. HTTP logs + gRPC traces) build the right exporter. - logredact: entering a group whose name is secret-bearing (e.g. WithGroup("authorization")) now masks every leaf in that subtree, matching how slog.Group("authorization", ...) is masked as a whole. * fix(observability): address review feedback on telemetry bootstrap - Telemetry setup failure no longer kills boot: Setup returns usable no-op providers alongside the error and main logs and continues with telemetry disabled, honoring the best-effort contract. - Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_* variants); unsupported values fall back to parentbased_traceidratio. - Attach node identity as semconv service.instance.id instead of the non-semconv node.name. - Rename opslog retention-scope log attrs to target_component/target_level so they no longer collide with the canonical component routing key, and tag those lines with component=opslog. - Fix stale levelGated comment casing; use WarnContext in the telemetry shutdown defer; document the LogValuer double-resolve on the redaction slow path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
169 lines
6.4 KiB
Go
169 lines
6.4 KiB
Go
// Package telemetry provides the OpenTelemetry SDK bootstrap for Silo: shared
|
|
// resource, tracer provider, logger provider, and W3C propagation. It
|
|
// deliberately does NOT build or register a MeterProvider — metrics stay on the
|
|
// existing Prometheus rail. Leaving the global MeterProvider as the built-in
|
|
// no-op is what prevents the trace instrumentation libraries from double-emitting
|
|
// metrics. See docs/superpowers/plans/2026-07-02-opentelemetry-observability.md.
|
|
package telemetry
|
|
|
|
import (
|
|
"math"
|
|
"os"
|
|
"strconv"
|
|
"strings"
|
|
)
|
|
|
|
// Protocol identifies the OTLP exporter wire protocol.
|
|
type Protocol string
|
|
|
|
const (
|
|
// ProtocolGRPC is the OTLP/gRPC protocol (default).
|
|
ProtocolGRPC Protocol = "grpc"
|
|
// ProtocolHTTP is the OTLP/HTTP+protobuf protocol.
|
|
ProtocolHTTP Protocol = "http/protobuf"
|
|
)
|
|
|
|
// Sampler identifies the head-sampling strategy (OTEL_TRACES_SAMPLER).
|
|
type Sampler string
|
|
|
|
const (
|
|
// SamplerAlwaysOn samples every trace.
|
|
SamplerAlwaysOn Sampler = "always_on"
|
|
// SamplerAlwaysOff samples nothing.
|
|
SamplerAlwaysOff Sampler = "always_off"
|
|
// SamplerTraceIDRatio samples by trace-id ratio regardless of the parent.
|
|
SamplerTraceIDRatio Sampler = "traceidratio"
|
|
// SamplerParentBasedAlwaysOn honors the parent decision, sampling roots.
|
|
SamplerParentBasedAlwaysOn Sampler = "parentbased_always_on"
|
|
// SamplerParentBasedAlwaysOff honors the parent decision, dropping roots.
|
|
SamplerParentBasedAlwaysOff Sampler = "parentbased_always_off"
|
|
// SamplerParentBasedTraceIDRatio honors the parent decision, sampling roots
|
|
// by trace-id ratio. This is the default.
|
|
SamplerParentBasedTraceIDRatio Sampler = "parentbased_traceidratio"
|
|
)
|
|
|
|
// defaultServiceName is used when OTEL_SERVICE_NAME is unset.
|
|
const defaultServiceName = "silo-server"
|
|
|
|
// defaultSamplerRatio is the parent-based trace-id ratio applied when
|
|
// OTEL_TRACES_SAMPLER_ARG is unset or unparseable.
|
|
const defaultSamplerRatio = 1.0
|
|
|
|
// Config is the fully-defaulted telemetry configuration parsed from the
|
|
// environment. It is cheap to construct and safe to build even when telemetry
|
|
// is disabled.
|
|
type Config struct {
|
|
// Enabled gates the entire feature. True when SILO_OTEL_ENABLED is truthy
|
|
// OR OTEL_EXPORTER_OTLP_ENDPOINT is set.
|
|
Enabled bool
|
|
|
|
// Endpoint is the OTLP collector endpoint (OTEL_EXPORTER_OTLP_ENDPOINT). It
|
|
// is used ONLY to decide Enabled; the exporters themselves read the endpoint
|
|
// (and all other OTEL_EXPORTER_OTLP_* knobs) directly from the environment,
|
|
// which remains the single source of truth for exporter wiring.
|
|
Endpoint string
|
|
// Protocol is the generic OTLP wire protocol (OTEL_EXPORTER_OTLP_PROTOCOL).
|
|
// It is the fallback for any signal without a signal-specific override.
|
|
Protocol Protocol
|
|
// TracesProtocol selects the trace exporter's wire protocol, honoring
|
|
// OTEL_EXPORTER_OTLP_TRACES_PROTOCOL and falling back to Protocol.
|
|
TracesProtocol Protocol
|
|
// LogsProtocol selects the log exporter's wire protocol, honoring
|
|
// OTEL_EXPORTER_OTLP_LOGS_PROTOCOL and falling back to Protocol.
|
|
LogsProtocol Protocol
|
|
|
|
// ServiceName populates the service.name resource attribute.
|
|
ServiceName string
|
|
// ServiceVersion populates the service.version resource attribute.
|
|
ServiceVersion string
|
|
// NodeID populates the service.instance.id resource attribute.
|
|
NodeID string
|
|
|
|
// Sampler is the head-sampling strategy (OTEL_TRACES_SAMPLER). Unrecognized
|
|
// or unsupported values (e.g. jaeger_remote) fall back to
|
|
// parentbased_traceidratio.
|
|
Sampler Sampler
|
|
// SamplerRatio is the trace-id-ratio sampling probability used by the
|
|
// ratio-based samplers (OTEL_TRACES_SAMPLER_ARG).
|
|
SamplerRatio float64
|
|
}
|
|
|
|
// LoadConfig parses the telemetry configuration from the environment. nodeID is
|
|
// the resolved node identity used for the service.instance.id resource
|
|
// attribute.
|
|
func LoadConfig(nodeID string) Config {
|
|
endpoint := strings.TrimSpace(os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT"))
|
|
enabled := truthy(os.Getenv("SILO_OTEL_ENABLED")) || endpoint != ""
|
|
|
|
serviceName := strings.TrimSpace(os.Getenv("OTEL_SERVICE_NAME"))
|
|
if serviceName == "" {
|
|
serviceName = defaultServiceName
|
|
}
|
|
|
|
// The generic protocol is the fallback; per-signal env vars override it for
|
|
// their own exporter so mixed collector setups (e.g. HTTP logs, gRPC traces)
|
|
// work as the OTLP spec prescribes.
|
|
protocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_PROTOCOL"), ProtocolGRPC)
|
|
tracesProtocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"), protocol)
|
|
logsProtocol := parseProtocol(os.Getenv("OTEL_EXPORTER_OTLP_LOGS_PROTOCOL"), protocol)
|
|
|
|
ratio := defaultSamplerRatio
|
|
if raw := strings.TrimSpace(os.Getenv("OTEL_TRACES_SAMPLER_ARG")); raw != "" {
|
|
// Accept only finite, non-negative values; clamp above 1 to 1.0 so a
|
|
// typo'd or +Inf arg means "sample everything" rather than silently
|
|
// falling through. NaN and -Inf fail the v >= 0 / IsInf checks.
|
|
if v, err := strconv.ParseFloat(raw, 64); err == nil && v >= 0 && !math.IsInf(v, 1) {
|
|
ratio = math.Min(v, 1.0)
|
|
}
|
|
}
|
|
|
|
return Config{
|
|
Enabled: enabled,
|
|
Endpoint: endpoint,
|
|
Protocol: protocol,
|
|
TracesProtocol: tracesProtocol,
|
|
LogsProtocol: logsProtocol,
|
|
ServiceName: serviceName,
|
|
ServiceVersion: strings.TrimSpace(os.Getenv("OTEL_SERVICE_VERSION")),
|
|
NodeID: nodeID,
|
|
Sampler: parseSampler(os.Getenv("OTEL_TRACES_SAMPLER")),
|
|
SamplerRatio: ratio,
|
|
}
|
|
}
|
|
|
|
// parseSampler maps an OTEL_TRACES_SAMPLER value to a Sampler, falling back to
|
|
// parentbased_traceidratio when the value is empty, unrecognized, or names a
|
|
// sampler this bootstrap does not support (e.g. jaeger_remote).
|
|
func parseSampler(raw string) Sampler {
|
|
switch s := Sampler(strings.ToLower(strings.TrimSpace(raw))); s {
|
|
case SamplerAlwaysOn, SamplerAlwaysOff, SamplerTraceIDRatio,
|
|
SamplerParentBasedAlwaysOn, SamplerParentBasedAlwaysOff:
|
|
return s
|
|
default:
|
|
return SamplerParentBasedTraceIDRatio
|
|
}
|
|
}
|
|
|
|
// parseProtocol maps an OTEL_EXPORTER_OTLP*_PROTOCOL value to a Protocol,
|
|
// returning fallback when the value is empty or unrecognized.
|
|
func parseProtocol(raw string, fallback Protocol) Protocol {
|
|
switch strings.ToLower(strings.TrimSpace(raw)) {
|
|
case "http/protobuf", "http":
|
|
return ProtocolHTTP
|
|
case "grpc":
|
|
return ProtocolGRPC
|
|
default:
|
|
return fallback
|
|
}
|
|
}
|
|
|
|
// truthy reports whether an env value should be treated as a boolean true.
|
|
func truthy(v string) bool {
|
|
switch strings.ToLower(strings.TrimSpace(v)) {
|
|
case "1", "true", "yes", "on":
|
|
return true
|
|
default:
|
|
return false
|
|
}
|
|
}
|