* feat(observability): OpenTelemetry logs+traces with secret redaction Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged. Bootstrap (internal/telemetry): - Setup() builds one shared resource, a TracerProvider (parent-based trace-id ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator from env. It installs NO MeterProvider — metrics stay on Prometheus, and the built-in no-op global MeterProvider keeps the trace instrumentation libs from double-emitting. Shutdown is deferred with a flush timeout. - Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the shared LevelVar and best-effort so a failing collector can't break the console or DB branches. stderr + opslog stay untouched. Secret redaction (internal/logredact): - A slog.Handler masks secret-keyed attributes (password, token, api_key, authorization, cookie, ...) — including .With-bound attrs, nested groups, secret-keyed group subtrees, and values behind a LogValuer — on the console and OTLP sinks, with a no-op fast path when a record has no secret keys. opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one marker list. Rotation is infra-managed (no custom file sink): container runtime for stderr, collector/backend for OTLP, opslog partition-pruning for the DB. Documented in docs/architecture/observability.md. Verification: go build ./..., go vet, gofmt -l — clean; go test ./internal/telemetry/ ./internal/logredact/ -race pass. AI-use disclosure: implemented with AI assistance (Claude Code), including adversarial reviews that hardened the bootstrap and fixed two redaction leak paths; reviewed by the author. * refactor(observability): slog context+component sweep, sloglint gate (phase 3) Part of #265. Builds on the OTel bootstrap + redaction commit. Standardizes every log call site onto the context-carrying slog variants so records correlate with the active OpenTelemetry trace, and locks the standard in with a machine gate so future code (human- or AI-authored) can't drift back. - Call-site sweep: converted the remaining slog.<Level>(...) calls to the slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope (background/init calls with no ctx are left as-is), across 183 files. Applied via a type-aware AST codemod. Log levels and message strings are preserved verbatim; a component attr (canonical per-package name) is added to direct package-level slog calls. Bound-logger calls keep their existing .With bindings. The main.go and telemetry package conversions rode with their file in the previous commit to keep each file within a single commit. - Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg, key-naming-case=snake, no-mixed-args. After the sweep all four report zero violations repo-wide (tests included), so make lint / CI now blocks any regression to the non-context form. The gate ships with the sweep because it cannot be green until the legacy sites are converted. Metrics remain on Prometheus; no behavior change to /metrics or Grafana. Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4 rules) 0 violations repo-wide; log levels verified unchanged. AI-use disclosure: implemented with AI assistance (Claude Code), including the codemod; reviewed by the author. * fix(observability): honor per-signal OTLP protocol and secret WithGroup names Two Codex review findings on PR #290: - telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector setups (e.g. HTTP logs + gRPC traces) build the right exporter. - logredact: entering a group whose name is secret-bearing (e.g. WithGroup("authorization")) now masks every leaf in that subtree, matching how slog.Group("authorization", ...) is masked as a whole. * fix(observability): address review feedback on telemetry bootstrap - Telemetry setup failure no longer kills boot: Setup returns usable no-op providers alongside the error and main logs and continues with telemetry disabled, honoring the best-effort contract. - Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_* variants); unsupported values fall back to parentbased_traceidratio. - Attach node identity as semconv service.instance.id instead of the non-semconv node.name. - Rename opslog retention-scope log attrs to target_component/target_level so they no longer collide with the canonical component routing key, and tag those lines with component=opslog. - Fix stale levelGated comment casing; use WarnContext in the telemetry shutdown defer; document the LogValuer double-resolve on the redaction slow path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
495 lines
13 KiB
Go
495 lines
13 KiB
Go
package handlers
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"log/slog"
|
|
"net/http"
|
|
"strconv"
|
|
"strings"
|
|
"time"
|
|
|
|
"github.com/gorilla/websocket"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/activitylog"
|
|
"github.com/Silo-Server/silo-server/internal/logstream"
|
|
"github.com/Silo-Server/silo-server/internal/opslog"
|
|
)
|
|
|
|
type AdminLogsHandler struct {
|
|
opsRepo *opslog.Repo
|
|
auditRepo *activitylog.Repo
|
|
streamHub *logstream.Hub
|
|
}
|
|
|
|
func NewAdminLogsHandler(opsRepo *opslog.Repo, auditRepo *activitylog.Repo, streamHub *logstream.Hub) *AdminLogsHandler {
|
|
return &AdminLogsHandler{opsRepo: opsRepo, auditRepo: auditRepo, streamHub: streamHub}
|
|
}
|
|
|
|
func (h *AdminLogsHandler) HandleListOperationalLogs(w http.ResponseWriter, r *http.Request) {
|
|
opts, err := parseOperationalLogOptionsFromRequest(r)
|
|
if err != nil {
|
|
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
|
return
|
|
}
|
|
|
|
result, err := h.opsRepo.List(r.Context(), opts)
|
|
if err != nil {
|
|
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to query operational logs")
|
|
return
|
|
}
|
|
writeJSON(w, http.StatusOK, result)
|
|
}
|
|
|
|
func (h *AdminLogsHandler) HandleListAuditLogs(w http.ResponseWriter, r *http.Request) {
|
|
opts, err := parseAuditLogOptionsFromRequest(r)
|
|
if err != nil {
|
|
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
|
return
|
|
}
|
|
|
|
result, err := h.auditRepo.List(r.Context(), opts)
|
|
if err != nil {
|
|
writeError(w, http.StatusInternalServerError, "internal_error", "Failed to query audit logs")
|
|
return
|
|
}
|
|
writeJSON(w, http.StatusOK, result)
|
|
}
|
|
|
|
func parseOperationalLogOptionsFromRequest(r *http.Request) (opslog.ListOptions, error) {
|
|
opts := opslog.ListOptions{
|
|
Level: strings.TrimSpace(r.URL.Query().Get("level")),
|
|
Component: strings.TrimSpace(r.URL.Query().Get("component")),
|
|
NodeID: strings.TrimSpace(r.URL.Query().Get("node_id")),
|
|
RequestID: strings.TrimSpace(r.URL.Query().Get("request_id")),
|
|
SessionID: strings.TrimSpace(r.URL.Query().Get("session_id")),
|
|
PlaybackSessionID: strings.TrimSpace(r.URL.Query().Get("playback_session_id")),
|
|
Query: strings.TrimSpace(r.URL.Query().Get("q")),
|
|
Cursor: strings.TrimSpace(r.URL.Query().Get("cursor")),
|
|
Limit: parseLimit(r, 100),
|
|
}
|
|
|
|
userID, err := parseOptionalIntQuery(r, "user_id")
|
|
if err != nil {
|
|
return opslog.ListOptions{}, err
|
|
}
|
|
opts.UserID = userID
|
|
|
|
if from, err := parseTimeQuery(r, "from"); err != nil {
|
|
return opslog.ListOptions{}, err
|
|
} else {
|
|
opts.From = from
|
|
}
|
|
if to, err := parseTimeQuery(r, "to"); err != nil {
|
|
return opslog.ListOptions{}, err
|
|
} else {
|
|
opts.To = to
|
|
}
|
|
|
|
return opts, nil
|
|
}
|
|
|
|
func parseAuditLogOptionsFromRequest(r *http.Request) (activitylog.ListOptions, error) {
|
|
opts := activitylog.ListOptions{
|
|
Method: strings.TrimSpace(r.URL.Query().Get("method")),
|
|
PathPrefix: strings.TrimSpace(r.URL.Query().Get("path_prefix")),
|
|
ClientIP: strings.TrimSpace(r.URL.Query().Get("client_ip")),
|
|
RequestID: strings.TrimSpace(r.URL.Query().Get("request_id")),
|
|
SessionID: strings.TrimSpace(r.URL.Query().Get("session_id")),
|
|
PlaybackSessionID: strings.TrimSpace(r.URL.Query().Get("playback_session_id")),
|
|
Cursor: strings.TrimSpace(r.URL.Query().Get("cursor")),
|
|
Limit: parseLimit(r, 100),
|
|
}
|
|
|
|
statusCode, err := parseOptionalIntQuery(r, "status_code")
|
|
if err != nil {
|
|
return activitylog.ListOptions{}, err
|
|
}
|
|
opts.StatusCode = statusCode
|
|
|
|
userID, err := parseOptionalIntQuery(r, "user_id")
|
|
if err != nil {
|
|
return activitylog.ListOptions{}, err
|
|
}
|
|
opts.UserID = userID
|
|
|
|
if from, err := parseTimeQuery(r, "from"); err != nil {
|
|
return activitylog.ListOptions{}, err
|
|
} else {
|
|
opts.From = from
|
|
}
|
|
if to, err := parseTimeQuery(r, "to"); err != nil {
|
|
return activitylog.ListOptions{}, err
|
|
} else {
|
|
opts.To = to
|
|
}
|
|
|
|
return opts, nil
|
|
}
|
|
|
|
func parseTimeQuery(r *http.Request, key string) (*time.Time, error) {
|
|
raw := strings.TrimSpace(r.URL.Query().Get(key))
|
|
if raw == "" {
|
|
return nil, nil
|
|
}
|
|
ts, err := time.Parse(time.RFC3339, raw)
|
|
if err != nil {
|
|
return nil, invalidQueryError(key)
|
|
}
|
|
value := ts.UTC()
|
|
return &value, nil
|
|
}
|
|
|
|
func parseOptionalIntQuery(r *http.Request, key string) (*int, error) {
|
|
raw := strings.TrimSpace(r.URL.Query().Get(key))
|
|
if raw == "" {
|
|
return nil, nil
|
|
}
|
|
value, err := strconv.Atoi(raw)
|
|
if err != nil {
|
|
return nil, invalidQueryError(key)
|
|
}
|
|
return &value, nil
|
|
}
|
|
|
|
func invalidQueryError(key string) error {
|
|
return &requestParseError{message: "Invalid " + key}
|
|
}
|
|
|
|
type requestParseError struct {
|
|
message string
|
|
}
|
|
|
|
func (e *requestParseError) Error() string {
|
|
return e.message
|
|
}
|
|
|
|
func parseLimit(r *http.Request, fallback int) int {
|
|
raw := strings.TrimSpace(r.URL.Query().Get("limit"))
|
|
if raw == "" {
|
|
return fallback
|
|
}
|
|
limit, err := strconv.Atoi(raw)
|
|
if err != nil || limit <= 0 {
|
|
return fallback
|
|
}
|
|
if limit > 200 {
|
|
return 200
|
|
}
|
|
return limit
|
|
}
|
|
|
|
func (h *AdminLogsHandler) HandleLogStreamWebSocket(w http.ResponseWriter, r *http.Request) {
|
|
if h.streamHub == nil {
|
|
http.Error(w, "log stream unavailable", http.StatusServiceUnavailable)
|
|
return
|
|
}
|
|
|
|
stream := logstream.Stream(strings.TrimSpace(r.URL.Query().Get("stream")))
|
|
if stream != logstream.StreamApp && stream != logstream.StreamAudit {
|
|
writeError(w, http.StatusBadRequest, "bad_request", "Invalid stream")
|
|
return
|
|
}
|
|
|
|
var (
|
|
appOpts opslog.ListOptions
|
|
auditOpts activitylog.ListOptions
|
|
err error
|
|
)
|
|
switch stream {
|
|
case logstream.StreamApp:
|
|
appOpts, err = parseOperationalLogOptionsFromRequest(r)
|
|
case logstream.StreamAudit:
|
|
auditOpts, err = parseAuditLogOptionsFromRequest(r)
|
|
}
|
|
if err != nil {
|
|
writeError(w, http.StatusBadRequest, "bad_request", err.Error())
|
|
return
|
|
}
|
|
|
|
conn, err := wsUpgrader.Upgrade(w, r, nil)
|
|
if err != nil {
|
|
return
|
|
}
|
|
defer conn.Close()
|
|
|
|
events, unsubscribe := h.streamHub.Subscribe(func(msg logstream.Message) bool {
|
|
return msg.Type == logstream.MessageTypeAppend && msg.Stream == stream
|
|
})
|
|
defer unsubscribe()
|
|
|
|
conn.SetReadDeadline(time.Now().Add(wsPingInterval + wsPongTimeout))
|
|
conn.SetPongHandler(func(string) error {
|
|
return conn.SetReadDeadline(time.Now().Add(wsPingInterval + wsPongTimeout))
|
|
})
|
|
|
|
done := make(chan struct{})
|
|
go func() {
|
|
defer close(done)
|
|
for {
|
|
if _, _, err := conn.ReadMessage(); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}()
|
|
|
|
buffered := drainBufferedMessages(events)
|
|
seenIDs := make(map[int64]struct{})
|
|
newestSnapshotID := int64(0)
|
|
|
|
switch stream {
|
|
case logstream.StreamApp:
|
|
if h.opsRepo == nil {
|
|
h.writeStreamError(conn, stream, "internal_error", "Operational log stream unavailable")
|
|
return
|
|
}
|
|
result, err := h.opsRepo.List(context.Background(), appOpts)
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "admin log stream operational snapshot failed", "component", "api", "error", err)
|
|
h.writeStreamError(conn, stream, "internal_error", "Failed to query operational logs")
|
|
return
|
|
}
|
|
if len(result.Entries) > 0 {
|
|
newestSnapshotID = result.Entries[0].ID
|
|
}
|
|
for _, entry := range result.Entries {
|
|
seenIDs[entry.ID] = struct{}{}
|
|
}
|
|
if err := writeSnapshotMessage(conn, stream, result.Entries, result.NextCursor); err != nil {
|
|
return
|
|
}
|
|
buffered = append(buffered, drainBufferedMessages(events)...)
|
|
for _, msg := range buffered {
|
|
entry, ok := decodeAppEntry(msg)
|
|
if !ok {
|
|
continue
|
|
}
|
|
if newestSnapshotID > 0 && entry.ID <= newestSnapshotID {
|
|
continue
|
|
}
|
|
if !matchesOperationalLog(appOpts, entry) {
|
|
continue
|
|
}
|
|
if _, ok := seenIDs[entry.ID]; ok {
|
|
continue
|
|
}
|
|
seenIDs[entry.ID] = struct{}{}
|
|
if err := conn.WriteJSON(msg); err != nil {
|
|
return
|
|
}
|
|
}
|
|
case logstream.StreamAudit:
|
|
if h.auditRepo == nil {
|
|
h.writeStreamError(conn, stream, "internal_error", "Audit log stream unavailable")
|
|
return
|
|
}
|
|
result, err := h.auditRepo.List(context.Background(), auditOpts)
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "admin log stream audit snapshot failed", "component", "api", "error", err)
|
|
h.writeStreamError(conn, stream, "internal_error", "Failed to query audit logs")
|
|
return
|
|
}
|
|
if len(result.Entries) > 0 {
|
|
newestSnapshotID = result.Entries[0].ID
|
|
}
|
|
for _, entry := range result.Entries {
|
|
seenIDs[entry.ID] = struct{}{}
|
|
}
|
|
if err := writeSnapshotMessage(conn, stream, result.Entries, result.NextCursor); err != nil {
|
|
return
|
|
}
|
|
buffered = append(buffered, drainBufferedMessages(events)...)
|
|
for _, msg := range buffered {
|
|
entry, ok := decodeAuditEntry(msg)
|
|
if !ok {
|
|
continue
|
|
}
|
|
if newestSnapshotID > 0 && entry.ID <= newestSnapshotID {
|
|
continue
|
|
}
|
|
if !matchesAuditLog(auditOpts, entry) {
|
|
continue
|
|
}
|
|
if _, ok := seenIDs[entry.ID]; ok {
|
|
continue
|
|
}
|
|
seenIDs[entry.ID] = struct{}{}
|
|
if err := conn.WriteJSON(msg); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}
|
|
|
|
ticker := time.NewTicker(wsPingInterval)
|
|
defer ticker.Stop()
|
|
|
|
for {
|
|
select {
|
|
case <-r.Context().Done():
|
|
return
|
|
case <-done:
|
|
return
|
|
case msg, ok := <-events:
|
|
if !ok {
|
|
return
|
|
}
|
|
switch stream {
|
|
case logstream.StreamApp:
|
|
entry, ok := decodeAppEntry(msg)
|
|
if !ok || !matchesOperationalLog(appOpts, entry) {
|
|
continue
|
|
}
|
|
if _, ok := seenIDs[entry.ID]; ok {
|
|
continue
|
|
}
|
|
seenIDs[entry.ID] = struct{}{}
|
|
case logstream.StreamAudit:
|
|
entry, ok := decodeAuditEntry(msg)
|
|
if !ok || !matchesAuditLog(auditOpts, entry) {
|
|
continue
|
|
}
|
|
if _, ok := seenIDs[entry.ID]; ok {
|
|
continue
|
|
}
|
|
seenIDs[entry.ID] = struct{}{}
|
|
}
|
|
if err := conn.WriteJSON(msg); err != nil {
|
|
if websocket.IsCloseError(err, websocket.CloseGoingAway, websocket.CloseNormalClosure) {
|
|
return
|
|
}
|
|
return
|
|
}
|
|
case <-ticker.C:
|
|
if err := conn.WriteControl(websocket.PingMessage, nil, time.Now().Add(5*time.Second)); err != nil {
|
|
return
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
func writeSnapshotMessage(conn *websocket.Conn, stream logstream.Stream, entries any, nextCursor string) error {
|
|
raw, err := json.Marshal(entries)
|
|
if err != nil {
|
|
return err
|
|
}
|
|
return conn.WriteJSON(logstream.Message{
|
|
Type: logstream.MessageTypeSnapshot,
|
|
Stream: stream,
|
|
Entries: raw,
|
|
NextCursor: nextCursor,
|
|
})
|
|
}
|
|
|
|
func (h *AdminLogsHandler) writeStreamError(conn *websocket.Conn, stream logstream.Stream, code, message string) {
|
|
_ = conn.WriteJSON(logstream.Message{
|
|
Type: logstream.MessageTypeError,
|
|
Stream: stream,
|
|
Code: code,
|
|
Message: message,
|
|
})
|
|
_ = conn.WriteControl(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseInternalServerErr, message), time.Now().Add(5*time.Second))
|
|
}
|
|
|
|
func drainBufferedMessages(events <-chan logstream.Message) []logstream.Message {
|
|
buffered := make([]logstream.Message, 0, 8)
|
|
for {
|
|
select {
|
|
case msg, ok := <-events:
|
|
if !ok {
|
|
return buffered
|
|
}
|
|
buffered = append(buffered, msg)
|
|
default:
|
|
return buffered
|
|
}
|
|
}
|
|
}
|
|
|
|
func decodeAppEntry(msg logstream.Message) (opslog.EntryRow, bool) {
|
|
var entry opslog.EntryRow
|
|
if err := json.Unmarshal(msg.Entry, &entry); err != nil {
|
|
return opslog.EntryRow{}, false
|
|
}
|
|
return entry, true
|
|
}
|
|
|
|
func decodeAuditEntry(msg logstream.Message) (activitylog.AuditEntry, bool) {
|
|
var entry activitylog.AuditEntry
|
|
if err := json.Unmarshal(msg.Entry, &entry); err != nil {
|
|
return activitylog.AuditEntry{}, false
|
|
}
|
|
return entry, true
|
|
}
|
|
|
|
func matchesOperationalLog(opts opslog.ListOptions, entry opslog.EntryRow) bool {
|
|
if opts.From != nil && entry.Timestamp.Before(*opts.From) {
|
|
return false
|
|
}
|
|
if opts.To != nil && entry.Timestamp.After(*opts.To) {
|
|
return false
|
|
}
|
|
if opts.Level != "" && entry.Level != strings.ToLower(opts.Level) {
|
|
return false
|
|
}
|
|
if opts.Component != "" && entry.Component != opts.Component {
|
|
return false
|
|
}
|
|
if opts.NodeID != "" && entry.NodeID != opts.NodeID {
|
|
return false
|
|
}
|
|
if opts.RequestID != "" && entry.RequestID != opts.RequestID {
|
|
return false
|
|
}
|
|
if opts.UserID != nil {
|
|
if entry.UserID == nil || *entry.UserID != *opts.UserID {
|
|
return false
|
|
}
|
|
}
|
|
if opts.SessionID != "" && entry.SessionID != opts.SessionID {
|
|
return false
|
|
}
|
|
if opts.PlaybackSessionID != "" && entry.PlaybackSessionID != opts.PlaybackSessionID {
|
|
return false
|
|
}
|
|
if opts.Query != "" && !strings.Contains(strings.ToLower(entry.Message), strings.ToLower(opts.Query)) {
|
|
return false
|
|
}
|
|
return true
|
|
}
|
|
|
|
func matchesAuditLog(opts activitylog.ListOptions, entry activitylog.AuditEntry) bool {
|
|
if opts.From != nil && entry.Timestamp.Before(*opts.From) {
|
|
return false
|
|
}
|
|
if opts.To != nil && entry.Timestamp.After(*opts.To) {
|
|
return false
|
|
}
|
|
if opts.Method != "" && entry.Method != strings.ToUpper(opts.Method) {
|
|
return false
|
|
}
|
|
if opts.StatusCode != nil && entry.StatusCode != *opts.StatusCode {
|
|
return false
|
|
}
|
|
if opts.PathPrefix != "" && !strings.HasPrefix(entry.Path, opts.PathPrefix) {
|
|
return false
|
|
}
|
|
if opts.ClientIP != "" && entry.ClientIP != opts.ClientIP {
|
|
return false
|
|
}
|
|
if opts.RequestID != "" && entry.RequestID != opts.RequestID {
|
|
return false
|
|
}
|
|
if opts.UserID != nil {
|
|
if entry.UserID == nil || *entry.UserID != *opts.UserID {
|
|
return false
|
|
}
|
|
}
|
|
if opts.SessionID != "" && entry.SessionID != opts.SessionID {
|
|
return false
|
|
}
|
|
if opts.PlaybackSessionID != "" && entry.PlaybackSessionID != opts.PlaybackSessionID {
|
|
return false
|
|
}
|
|
return true
|
|
}
|