Files
silo-server/internal/audiobooks/abs/smart_collections_handler.go
203a18ae83 feat(observability): OpenTelemetry logs+traces with secret redaction and slog standardization (#290)
* feat(observability): OpenTelemetry logs+traces with secret redaction

Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing
stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with
no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged.

Bootstrap (internal/telemetry):
- Setup() builds one shared resource, a TracerProvider (parent-based trace-id
  ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator
  from env. It installs NO MeterProvider — metrics stay on Prometheus, and the
  built-in no-op global MeterProvider keeps the trace instrumentation libs from
  double-emitting. Shutdown is deferred with a flush timeout.
- Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the
  shared LevelVar and best-effort so a failing collector can't break the console
  or DB branches. stderr + opslog stay untouched.

Secret redaction (internal/logredact):
- A slog.Handler masks secret-keyed attributes (password, token, api_key,
  authorization, cookie, ...) — including .With-bound attrs, nested groups,
  secret-keyed group subtrees, and values behind a LogValuer — on the console
  and OTLP sinks, with a no-op fast path when a record has no secret keys.
  opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one
  marker list.

Rotation is infra-managed (no custom file sink): container runtime for stderr,
collector/backend for OTLP, opslog partition-pruning for the DB. Documented in
docs/architecture/observability.md.

Verification: go build ./..., go vet, gofmt -l — clean; go test
./internal/telemetry/ ./internal/logredact/ -race pass.

AI-use disclosure: implemented with AI assistance (Claude Code), including
adversarial reviews that hardened the bootstrap and fixed two redaction leak
paths; reviewed by the author.

* refactor(observability): slog context+component sweep, sloglint gate (phase 3)

Part of #265. Builds on the OTel bootstrap + redaction commit.

Standardizes every log call site onto the context-carrying slog variants so
records correlate with the active OpenTelemetry trace, and locks the standard
in with a machine gate so future code (human- or AI-authored) can't drift back.

- Call-site sweep: converted the remaining slog.<Level>(...) calls to the
  slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope
  (background/init calls with no ctx are left as-is), across 183 files. Applied
  via a type-aware AST codemod. Log levels and message strings are preserved
  verbatim; a component attr (canonical per-package name) is added to direct
  package-level slog calls. Bound-logger calls keep their existing .With
  bindings. The main.go and telemetry package conversions rode with their file
  in the previous commit to keep each file within a single commit.
- Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg,
  key-naming-case=snake, no-mixed-args. After the sweep all four report zero
  violations repo-wide (tests included), so make lint / CI now blocks any
  regression to the non-context form. The gate ships with the sweep because it
  cannot be green until the legacy sites are converted.

Metrics remain on Prometheus; no behavior change to /metrics or Grafana.

Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4
rules) 0 violations repo-wide; log levels verified unchanged.

AI-use disclosure: implemented with AI assistance (Claude Code), including the
codemod; reviewed by the author.

* fix(observability): honor per-signal OTLP protocol and secret WithGroup names

Two Codex review findings on PR #290:

- telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the
  generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector
  setups (e.g. HTTP logs + gRPC traces) build the right exporter.
- logredact: entering a group whose name is secret-bearing (e.g.
  WithGroup("authorization")) now masks every leaf in that subtree,
  matching how slog.Group("authorization", ...) is masked as a whole.

* fix(observability): address review feedback on telemetry bootstrap

- Telemetry setup failure no longer kills boot: Setup returns usable
  no-op providers alongside the error and main logs and continues with
  telemetry disabled, honoring the best-effort contract.
- Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_*
  variants); unsupported values fall back to parentbased_traceidratio.
- Attach node identity as semconv service.instance.id instead of the
  non-semconv node.name.
- Rename opslog retention-scope log attrs to target_component/target_level
  so they no longer collide with the canonical component routing key, and
  tag those lines with component=opslog.
- Fix stale levelGated comment casing; use WarnContext in the telemetry
  shutdown defer; document the LogValuer double-resolve on the redaction
  slow path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-09 08:53:52 -04:00

417 lines
13 KiB
Go

package abs
import (
"encoding/json"
"errors"
"io"
"log/slog"
"net/http"
"github.com/oklog/ulid/v2"
"github.com/Silo-Server/silo-server/internal/audiobooks/smartcoll"
"github.com/Silo-Server/silo-server/internal/models"
)
// smartCollectionBody is the JSON body for POST and PATCH
// /me/smart-collections[/{id}]. Pointer fields support partial PATCH.
type smartCollectionBody struct {
Name *string `json:"name"`
Description *string `json:"description"`
Color *string `json:"color"`
IsPublic *bool `json:"isPublic"`
IsPinned *bool `json:"isPinned"`
QueryDef *smartcoll.QueryDefinition `json:"query_def"`
}
// handleCreateSmartCollection — POST /me/smart-collections.
func (h *Handler) handleCreateSmartCollection(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
http.Error(w, "smart collection store unavailable", http.StatusServiceUnavailable)
return
}
var body smartCollectionBody
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil {
http.Error(w, "invalid body", http.StatusBadRequest)
return
}
if body.Name == nil || *body.Name == "" {
http.Error(w, "name required", http.StatusBadRequest)
return
}
c := SmartCollection{
ID: ulid.Make().String(),
UserID: a.UserID,
ProfileID: a.ProfileID,
Name: *body.Name,
}
if body.Description != nil {
c.Description = *body.Description
}
if body.Color != nil {
c.Color = *body.Color
}
if body.IsPublic != nil {
c.IsPublic = *body.IsPublic
}
if body.IsPinned != nil {
c.IsPinned = *body.IsPinned
}
qd := smartcoll.QueryDefinition{}
if body.QueryDef != nil {
qd = *body.QueryDef
}
qd = qd.Normalize()
if err := qd.Validate(true); err != nil {
http.Error(w, "invalid query_def: "+err.Error(), http.StatusBadRequest)
return
}
qdBytes, err := json.Marshal(qd)
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection marshal query_def failed", "component", "audiobooks", "err", err)
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
return
}
c.QueryDef = qdBytes
if err := h.deps.SmartCollectionStore.CreateSmartCollection(r.Context(), c); err != nil {
slog.ErrorContext(r.Context(), "abs smart collection create failed", "component", "audiobooks", "err", err, "user", a.UserID)
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
return
}
persisted, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), c.ID)
if errors.Is(err, ErrNotFound) || err != nil {
persisted = c
}
writeJSON(w, http.StatusOK, smartCollectionToABS(persisted))
}
func (h *Handler) handleListSmartCollections(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
writeJSON(w, http.StatusOK, map[string]any{"items": []any{}})
return
}
rows, err := h.deps.SmartCollectionStore.ListUserSmartCollections(r.Context(), a.UserID, a.ProfileID)
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection list failed", "component", "audiobooks", "err", err, "user", a.UserID)
http.Error(w, "smart collection list failed", http.StatusInternalServerError)
return
}
out := make([]map[string]any, 0, len(rows))
for _, c := range rows {
out = append(out, smartCollectionToABS(c))
}
writeJSON(w, http.StatusOK, map[string]any{"items": out})
}
func (h *Handler) handleGetSmartCollection(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), chiURLID(r))
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID) && !c.IsPublic) {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection get failed", "component", "audiobooks", "err", err)
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
return
}
writeJSON(w, http.StatusOK, smartCollectionToABS(c))
}
func (h *Handler) handleUpdateSmartCollection(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
id := chiURLID(r)
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID)) {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection get-for-update failed", "component", "audiobooks", "err", err, "id", id)
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
return
}
var body smartCollectionBody
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil {
http.Error(w, "invalid body", http.StatusBadRequest)
return
}
if body.Name != nil {
c.Name = *body.Name
}
if body.Description != nil {
c.Description = *body.Description
}
if body.Color != nil {
c.Color = *body.Color
}
if body.IsPublic != nil {
c.IsPublic = *body.IsPublic
}
if body.IsPinned != nil {
c.IsPinned = *body.IsPinned
}
if body.QueryDef != nil {
qd := body.QueryDef.Normalize()
if err := qd.Validate(true); err != nil {
http.Error(w, "invalid query_def: "+err.Error(), http.StatusBadRequest)
return
}
qdBytes, mErr := json.Marshal(qd)
if mErr != nil {
slog.ErrorContext(r.Context(), "abs smart collection marshal query_def failed", "component", "audiobooks", "err", mErr)
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
return
}
c.QueryDef = qdBytes
}
if err := h.deps.SmartCollectionStore.UpdateSmartCollection(r.Context(), c); err != nil {
slog.ErrorContext(r.Context(), "abs smart collection update failed", "component", "audiobooks", "err", err, "id", id)
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
return
}
persisted, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
if err != nil {
persisted = c
}
writeJSON(w, http.StatusOK, smartCollectionToABS(persisted))
}
func (h *Handler) handleDeleteSmartCollection(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
id := chiURLID(r)
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID)) {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection get-for-delete failed", "component", "audiobooks", "err", err, "id", id)
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
return
}
if err := h.deps.SmartCollectionStore.DeleteSmartCollection(r.Context(), id); err != nil {
slog.ErrorContext(r.Context(), "abs smart collection delete failed", "component", "audiobooks", "err", err, "id", id)
http.Error(w, "smart collection delete failed", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusNoContent)
}
// handleSmartCollectionItems — GET /me/smart-collections/{id}/items.
// Evaluates the collection's query_def against the audiobook catalog
// and returns a paged envelope. When the caller is the owner, per-user
// state is hydrated; non-owner viewing a public collection sees
// personalized rules silently dropped.
func (h *Handler) handleSmartCollectionItems(w http.ResponseWriter, r *http.Request) {
a, ok := absAuthFrom(r)
if !ok || a.UserID == "" {
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
if h.deps.SmartCollectionStore == nil {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
id := chiURLID(r)
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID) && !c.IsPublic) {
http.Error(w, "smart collection not found", http.StatusNotFound)
return
}
if err != nil {
slog.ErrorContext(r.Context(), "abs smart collection items get failed", "component", "audiobooks", "err", err, "id", id)
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
return
}
var qd smartcoll.QueryDefinition
if len(c.QueryDef) > 0 {
if uErr := json.Unmarshal(c.QueryDef, &qd); uErr != nil {
slog.ErrorContext(r.Context(), "abs smart collection invalid stored query_def", "component", "audiobooks", "err", uErr, "id", id)
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
return
}
}
qd = qd.Normalize()
limit, page := readPagedQuery(r, 30)
if r.URL.Query().Get("limit") == "" && qd.Limit != nil && *qd.Limit > 0 {
limit = *qd.Limit
}
access, err := h.accessFilterForAuth(r.Context(), a)
if err != nil {
http.Error(w, "resolve access: "+err.Error(), http.StatusForbidden)
return
}
allLibs, err := h.deps.MediaStore.ListAudiobookLibraries(r.Context(), access)
if err != nil {
slog.WarnContext(r.Context(), "abs smart collection libraries fetch failed", "component", "audiobooks", "err", err, "id", id)
allLibs = nil
}
libByID := make(map[int64]AudiobookLibrary, len(allLibs))
for _, lib := range allLibs {
libByID[lib.ID] = lib
}
var targetLibs []AudiobookLibrary
if len(qd.LibraryIDs) > 0 {
for _, lid := range qd.LibraryIDs {
if lib, ok := libByID[lid]; ok {
targetLibs = append(targetLibs, lib)
}
}
} else {
targetLibs = allLibs
}
owner := sameABSPrincipal(a, c.UserID, c.ProfileID)
progressByID := map[string]ProgressRow{}
bookmarkCountByID := map[string]int{}
if owner {
if h.deps.ProgressStore != nil {
if rows, perr := h.deps.ProgressStore.ListProgressForAudiobooks(r.Context(), a.UserID, a.ProfileID, 10000); perr == nil {
for _, p := range rows {
progressByID[p.ContentID] = p
}
}
}
if h.deps.BookmarkStore != nil {
if counts, berr := h.deps.BookmarkStore.CountByUser(r.Context(), a.UserID, a.ProfileID); berr == nil {
bookmarkCountByID = counts
}
}
}
candidates := make([]smartcoll.Candidate, 0, 256)
for _, lib := range targetLibs {
items, _, lerr := h.deps.MediaStore.ListAudiobooks(r.Context(), lib.ID, 0, 0, access, Filter{})
if lerr != nil {
slog.WarnContext(r.Context(), "abs smart collection list-audiobooks failed", "component", "audiobooks", "err", lerr, "library", lib.ID)
continue
}
for _, mi := range items {
cand := smartcoll.Candidate{Item: siloItemToSmartcollItem(mi)}
if owner {
if p, ok := progressByID[mi.ContentID]; ok {
cand.IsFinished = p.IsFinished
cand.ProgressPct = float32(p.ProgressPct)
cand.CurrentSeconds = int(p.CurrentSeconds)
cand.LastPlayedAt = p.UpdatedAt
}
cand.BookmarkCount = bookmarkCountByID[mi.ContentID]
}
candidates = append(candidates, cand)
}
}
matched := smartcoll.Evaluate(r.Context(), qd, candidates, smartcoll.EvaluateOptions{
AllowPersonalized: owner,
UserSeed: a.UserID + ":" + c.ID,
})
total := len(matched)
start := page * limit
if start > total {
start = total
}
end := start + limit
if end > total {
end = total
}
pageSlice := matched[start:end]
libDefault := h.resolveDefaultLibrary(r.Context(), access)
libDefaultID := audiobookLibraryID(libDefault)
results := make([]map[string]any, 0, len(pageSlice))
for _, cand := range pageSlice {
entry := map[string]any{
"id": cand.Item.ID,
"libraryId": libDefaultID,
"media": map[string]any{
"metadata": map[string]any{"title": cand.Item.Title},
},
}
results = append(results, entry)
}
writeJSON(w, http.StatusOK, pagedEnvelope(results, total, limit, page, qd.Sort.Field, qd.Sort.Order == "desc", "", false, ""))
}
// siloItemToSmartcollItem maps a silo *models.MediaItem into the
// audiobook-domain Item shape the smartcoll evaluator walks.
func siloItemToSmartcollItem(mi *models.MediaItem) smartcoll.Item {
if mi == nil {
return smartcoll.Item{}
}
it := smartcoll.Item{
ID: mi.ContentID,
Title: mi.Title,
Genres: mi.Genres,
Year: mi.Year,
Language: mi.OriginalLanguage,
DurationSeconds: mi.Runtime,
}
for _, p := range mi.People {
switch p.Kind {
case models.PersonKindAuthor:
it.Authors = append(it.Authors, p.Name)
case models.PersonKindNarrator:
it.Narrators = append(it.Narrators, p.Name)
}
}
for _, s := range mi.AudiobookSeries {
it.Series = append(it.Series, s.Name)
}
if len(mi.Studios) > 0 {
it.Publisher = mi.Studios[0]
}
if mi.RatingIMDB != nil {
it.Rating = *mi.RatingIMDB
}
if mi.AddedAt != nil {
it.AddedAt = *mi.AddedAt
}
return it
}