Files
silo-server/internal/proxy/server.go
T
203a18ae83 feat(observability): OpenTelemetry logs+traces with secret redaction and slog standardization (#290)
* feat(observability): OpenTelemetry logs+traces with secret redaction

Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing
stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with
no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged.

Bootstrap (internal/telemetry):
- Setup() builds one shared resource, a TracerProvider (parent-based trace-id
  ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator
  from env. It installs NO MeterProvider — metrics stay on Prometheus, and the
  built-in no-op global MeterProvider keeps the trace instrumentation libs from
  double-emitting. Shutdown is deferred with a flush timeout.
- Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the
  shared LevelVar and best-effort so a failing collector can't break the console
  or DB branches. stderr + opslog stay untouched.

Secret redaction (internal/logredact):
- A slog.Handler masks secret-keyed attributes (password, token, api_key,
  authorization, cookie, ...) — including .With-bound attrs, nested groups,
  secret-keyed group subtrees, and values behind a LogValuer — on the console
  and OTLP sinks, with a no-op fast path when a record has no secret keys.
  opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one
  marker list.

Rotation is infra-managed (no custom file sink): container runtime for stderr,
collector/backend for OTLP, opslog partition-pruning for the DB. Documented in
docs/architecture/observability.md.

Verification: go build ./..., go vet, gofmt -l — clean; go test
./internal/telemetry/ ./internal/logredact/ -race pass.

AI-use disclosure: implemented with AI assistance (Claude Code), including
adversarial reviews that hardened the bootstrap and fixed two redaction leak
paths; reviewed by the author.

* refactor(observability): slog context+component sweep, sloglint gate (phase 3)

Part of #265. Builds on the OTel bootstrap + redaction commit.

Standardizes every log call site onto the context-carrying slog variants so
records correlate with the active OpenTelemetry trace, and locks the standard
in with a machine gate so future code (human- or AI-authored) can't drift back.

- Call-site sweep: converted the remaining slog.<Level>(...) calls to the
  slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope
  (background/init calls with no ctx are left as-is), across 183 files. Applied
  via a type-aware AST codemod. Log levels and message strings are preserved
  verbatim; a component attr (canonical per-package name) is added to direct
  package-level slog calls. Bound-logger calls keep their existing .With
  bindings. The main.go and telemetry package conversions rode with their file
  in the previous commit to keep each file within a single commit.
- Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg,
  key-naming-case=snake, no-mixed-args. After the sweep all four report zero
  violations repo-wide (tests included), so make lint / CI now blocks any
  regression to the non-context form. The gate ships with the sweep because it
  cannot be green until the legacy sites are converted.

Metrics remain on Prometheus; no behavior change to /metrics or Grafana.

Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4
rules) 0 violations repo-wide; log levels verified unchanged.

AI-use disclosure: implemented with AI assistance (Claude Code), including the
codemod; reviewed by the author.

* fix(observability): honor per-signal OTLP protocol and secret WithGroup names

Two Codex review findings on PR #290:

- telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the
  generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector
  setups (e.g. HTTP logs + gRPC traces) build the right exporter.
- logredact: entering a group whose name is secret-bearing (e.g.
  WithGroup("authorization")) now masks every leaf in that subtree,
  matching how slog.Group("authorization", ...) is masked as a whole.

* fix(observability): address review feedback on telemetry bootstrap

- Telemetry setup failure no longer kills boot: Setup returns usable
  no-op providers alongside the error and main logs and continues with
  telemetry disabled, honoring the best-effort contract.
- Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_*
  variants); unsupported values fall back to parentbased_traceidratio.
- Attach node identity as semconv service.instance.id instead of the
  non-semconv node.name.
- Rename opslog retention-scope log attrs to target_component/target_level
  so they no longer collide with the canonical component routing key, and
  tag those lines with component=opslog.
- Fix stale levelGated comment casing; use WarnContext in the telemetry
  shutdown defer; document the LogValuer double-resolve on the redaction
  slow path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 08:53:52 -04:00

371 lines
13 KiB
Go

package proxy
import (
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"strconv"
"strings"
"time"
"github.com/go-chi/chi/v5"
"github.com/go-chi/cors"
"github.com/Silo-Server/silo-server/internal/nodeconfig"
"github.com/Silo-Server/silo-server/internal/nodesessions"
"github.com/Silo-Server/silo-server/internal/playback"
"github.com/Silo-Server/silo-server/internal/streamtoken"
)
// Server is the HTTP handler for proxy mode.
type Server struct {
watcher *nodeconfig.Watcher
tracker *nodesessions.Tracker
httpClient *http.Client
egress *egressMeter
}
// NewServer creates a new proxy server backed by a config watcher and session
// tracker.
func NewServer(watcher *nodeconfig.Watcher, tracker *nodesessions.Tracker) *Server {
return &Server{
watcher: watcher,
tracker: tracker,
// No overall timeout — stream bodies are long-lived. Hung nodes are
// bounded by the transport's response-header timeout instead.
httpClient: &http.Client{Transport: newStreamTransport()},
egress: newEgressMeter(),
}
}
// newStreamTransport tunes the proxy→transcode-node connection pool. Many
// concurrent viewers fan their segment fetches through one proxy→node pair,
// and Go's default of 2 idle connections per host causes constant connection
// churn (and TLS re-handshakes) under load. The response-header timeout
// bounds requests to a hung node; the longest legitimate server-side wait is
// the 30s manifest-readiness poll on the transcode node.
func newStreamTransport() *http.Transport {
t := http.DefaultTransport.(*http.Transport).Clone()
t.MaxIdleConns = 128
t.MaxIdleConnsPerHost = 32
t.ResponseHeaderTimeout = 60 * time.Second
return t
}
// Handler returns the chi.Router with all proxy routes mounted.
func (s *Server) Handler() http.Handler {
r := chi.NewRouter()
// hls.js uses XHR for manifest/segment fetches which are subject to
// CORS when the proxy runs on a different origin than the web app.
r.Use(cors.Handler(cors.Options{
AllowedOrigins: []string{"*"},
AllowedMethods: []string{"GET", "HEAD", "OPTIONS"},
AllowedHeaders: []string{"Accept", "Authorization", "Content-Type", "Range"},
MaxAge: 86400,
}))
r.Get("/api/v1/health", s.handleHealth)
r.Group(func(r chi.Router) {
// Everything under /stream counts toward the node's measured
// egress bandwidth.
r.Use(s.meterEgress)
r.Head("/stream/direct/{token}", s.handleDirectPlay)
r.Get("/stream/direct/{token}", s.handleDirectPlay)
r.Head("/stream/remux/{token}", s.handleRemux)
r.Get("/stream/remux/{token}", s.handleRemux)
r.Head("/stream/transcode/{token}/master.m3u8", s.handleTranscodeManifest)
r.Get("/stream/transcode/{token}/master.m3u8", s.handleTranscodeManifest)
r.Get("/stream/transcode/{token}/segment/{name}", s.handleTranscodeSegment)
r.Get("/stream/subtitles/{token}/{track}/fonts", s.handleSubtitleFonts)
r.Get("/stream/subtitles/{token}/{track}", s.handleSubtitle)
})
// Admin routes — bearer-auth protected.
r.Group(func(r chi.Router) {
r.Use(s.requireBearer)
r.Post("/admin/force-reload", s.handleForceReload)
r.Get("/status", s.handleStatus)
})
return r
}
type healthResponse struct {
Status string `json:"status"`
ActiveJobs int `json:"active_jobs"`
EgressKbps int `json:"egress_kbps"`
}
func (s *Server) handleHealth(w http.ResponseWriter, _ *http.Request) {
activeJobs := 0
if s.tracker != nil {
activeJobs = s.tracker.ActiveCount()
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(healthResponse{
Status: "ok",
ActiveJobs: activeJobs,
EgressKbps: s.egress.RateKbps(),
})
}
// requireBearer checks Authorization: Bearer {secret} for admin endpoints.
func (s *Server) requireBearer(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
cfg := s.watcher.Config()
auth := r.Header.Get("Authorization")
if !strings.HasPrefix(auth, "Bearer ") || strings.TrimPrefix(auth, "Bearer ") != cfg.Auth.JWTSecret {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
next.ServeHTTP(w, r)
})
}
// verifyToken extracts and validates the stream token from the URL.
func (s *Server) verifyToken(w http.ResponseWriter, r *http.Request) *streamtoken.Claims {
cfg := s.watcher.Config()
tokenStr := chi.URLParam(r, "token")
claims, err := streamtoken.Verify(tokenStr, cfg.Auth.JWTSecret)
if err != nil {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return nil
}
return claims
}
func (s *Server) handleDirectPlay(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
info := sessionInfo(s.tracker, claims, "direct_play")
s.tracker.Track(r.Context(), info)
defer s.tracker.Remove(r.Context(), claims.SessionID)
http.ServeFile(w, r, claims.MediaPath)
}
func (s *Server) handleRemux(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
info := sessionInfo(s.tracker, claims, "remux")
s.tracker.Track(r.Context(), info)
defer s.tracker.Remove(r.Context(), claims.SessionID)
seekSeconds := 0.0
if seekStr := r.URL.Query().Get("seek"); seekStr != "" {
if v, err := strconv.ParseFloat(seekStr, 64); err == nil {
seekSeconds = v
}
}
_ = playback.ServeRemux(w, r, claims.MediaPath, "mp4", seekSeconds, claims.TranscodeAudio, claims.AudioTrackIndex, claims.DVProfile)
}
func (s *Server) handleTranscodeManifest(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
s.touchTranscodeSession(r, claims)
s.proxyToTranscodeNode(w, r, claims, "/transcode/"+claims.SessionID+"/master.m3u8")
}
func (s *Server) handleTranscodeSegment(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
s.touchTranscodeSession(r, claims)
name := chi.URLParam(r, "name")
s.proxyToTranscodeNode(w, r, claims, "/transcode/"+claims.SessionID+"/segment/"+name)
}
// touchTranscodeSession keeps HLS sessions visible in the active stream count.
// Unlike direct play and remux, transcode playback reaches the proxy as many
// short manifest/segment requests, so the session is tracked by recent
// activity instead of request lifetime.
func (s *Server) touchTranscodeSession(r *http.Request, claims *streamtoken.Claims) {
s.tracker.Touch(r.Context(), sessionInfo(s.tracker, claims, "transcode"))
}
// sessionInfo builds the node-session tracker record for a verified token,
// copying the numeric ownership keys the node-session tracker needs.
func sessionInfo(tr *nodesessions.Tracker, claims *streamtoken.Claims, kind string) nodesessions.SessionInfo {
return nodesessions.SessionInfo{
SessionID: claims.SessionID,
NodeURL: tr.NodeURL(),
NodeName: tr.NodeName(),
Type: kind,
StartedAt: time.Now().UTC().Format(time.RFC3339),
AuthUserID: claims.UserID,
ProfileID: claims.ProfileID,
MediaFileID: claims.MediaFileID,
}
}
func (s *Server) handleSubtitle(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
cfg := s.watcher.Config()
trackParam := chi.URLParam(r, "track")
trackIndex, requestedFormat, err := playback.ParseSubtitleTrackParam(trackParam)
if err != nil {
http.Error(w, "invalid subtitle index", http.StatusBadRequest)
return
}
// When the URL requests SUP format (e.g. /subtitles/{token}/2.sup),
// stream the PGS track as a raw .sup elementary stream for client-side
// bitmap rendering (libpgs). Unlike the buffered text paths below, this
// streams ffmpeg output directly: the track can be large and the client
// renders progressively as data arrives.
if requestedFormat == "sup" {
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Cache-Control", "no-store")
w.WriteHeader(http.StatusOK)
err := playback.StreamExtractSubtitle(r.Context(), playback.StreamExtractOpts{
InputPath: claims.MediaPath,
TrackIndex: trackIndex,
SourceCodec: "hdmv_pgs_subtitle", // .sup URLs are only generated for PGS tracks
FFmpegPath: cfg.Playback.FFmpegPath,
Writer: w,
})
if err != nil && r.Context().Err() == nil {
// Headers already committed — log and let the client see a
// truncated response.
slog.ErrorContext(r.Context(), "stream subtitle (sup)", "component", "proxy", "error", err, "track", trackIndex,
"path", claims.MediaPath, "playback_session_id", claims.SessionID)
}
return
}
// When the URL requests ASS format (e.g. /subtitles/{token}/2.ass),
// extract as raw ASS to preserve styling for client-side rendering.
if requestedFormat == "ass" {
data, err := playback.ExtractSubtitleWithFormat(r.Context(), claims.MediaPath, trackIndex, "ass", cfg.Playback.FFmpegPath)
if err != nil {
slog.ErrorContext(r.Context(), "extract subtitle (ass)", "component", "proxy", "error", err, "track", trackIndex, "path", claims.MediaPath, "playback_session_id", claims.SessionID)
http.Error(w, "subtitle extraction failed", http.StatusInternalServerError)
return
}
playback.ServeSubtitle(w, data, "ass")
return
}
data, format, err := playback.ExtractSubtitle(r.Context(), claims.MediaPath, trackIndex, cfg.Playback.FFmpegPath)
if err != nil {
slog.ErrorContext(r.Context(), "extract subtitle", "component", "proxy", "error", err, "track", trackIndex, "path", claims.MediaPath, "playback_session_id", claims.SessionID)
http.Error(w, "subtitle extraction failed", http.StatusInternalServerError)
return
}
vtt, err := playback.ConvertToVTT(data, format)
if err != nil {
slog.ErrorContext(r.Context(), "convert to vtt", "component", "proxy", "error", err, "playback_session_id", claims.SessionID)
http.Error(w, "subtitle conversion failed", http.StatusInternalServerError)
return
}
playback.ServeSubtitle(w, vtt, "vtt")
}
func (s *Server) handleSubtitleFonts(w http.ResponseWriter, r *http.Request) {
claims := s.verifyToken(w, r)
if claims == nil {
return
}
cfg := s.watcher.Config()
trackParam := chi.URLParam(r, "track")
trackIndex, _, err := playback.ParseSubtitleTrackParam(trackParam)
if err != nil {
http.Error(w, "invalid subtitle index", http.StatusBadRequest)
return
}
fonts, err := playback.ExtractAttachedSubtitleFonts(r.Context(), claims.MediaPath, cfg.Playback.FFmpegPath)
if err != nil {
slog.ErrorContext(r.Context(), "extract subtitle fonts", "component", "proxy", "error", err, "track", trackIndex, "path", claims.MediaPath, "playback_session_id", claims.SessionID)
http.Error(w, "subtitle font extraction failed", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Cache-Control", "no-store")
if err := json.NewEncoder(w).Encode(playback.EncodeSubtitleFontBundle(fonts)); err != nil {
slog.WarnContext(r.Context(), "subtitle font response encode failed", "component", "proxy", "error", err, "playback_session_id", claims.SessionID)
}
}
// proxyToTranscodeNode forwards the request to the transcode node specified in the claims.
func (s *Server) proxyToTranscodeNode(w http.ResponseWriter, r *http.Request, claims *streamtoken.Claims, path string) {
cfg := s.watcher.Config()
if claims.TranscodeNode == "" {
http.Error(w, "no transcode node in token", http.StatusBadRequest)
return
}
targetURL := claims.TranscodeNode + path
if rawQuery := r.URL.RawQuery; rawQuery != "" {
targetURL = fmt.Sprintf("%s?%s", targetURL, rawQuery)
}
req, err := http.NewRequestWithContext(r.Context(), r.Method, targetURL, nil)
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
req.Header.Set("Authorization", "Bearer "+cfg.Auth.JWTSecret)
// Forward the verified stream token so the transcode node can self-reconstruct
// a lost session after its OWN restart: the token carries the full byte-affecting
// recipe, so the node can re-spawn ffmpeg seeked to the requested segment instead
// of 404ing (the integrated server already does this from the same token). The
// node re-verifies the token independently before trusting it.
if token := chi.URLParam(r, "token"); token != "" {
req.Header.Set("X-Silo-Stream-Token", token)
}
resp, err := s.httpClient.Do(req)
if err != nil {
slog.ErrorContext(r.Context(), "proxy to transcode node", "component", "proxy", "error", err, "url", targetURL, "playback_session_id", claims.SessionID)
http.Error(w, "transcode node unavailable", http.StatusBadGateway)
return
}
defer resp.Body.Close()
// Copy response headers
for k, vv := range resp.Header {
for _, v := range vv {
w.Header().Add(k, v)
}
}
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}
func (s *Server) handleForceReload(w http.ResponseWriter, r *http.Request) {
if err := s.watcher.ForceReload(r.Context()); err != nil {
http.Error(w, "reload failed: "+err.Error(), http.StatusInternalServerError)
return
}
slog.InfoContext(r.Context(), "proxy force reload completed", slog.String("component", "proxy"))
w.WriteHeader(http.StatusNoContent)
}
type statusResponse struct {
ActiveSessions int `json:"active_sessions"`
}
func (s *Server) handleStatus(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(statusResponse{
ActiveSessions: s.tracker.ActiveCount(),
})
}