* feat(observability): OpenTelemetry logs+traces with secret redaction Part of #265. Adds opt-in OpenTelemetry (logs + traces) alongside the existing stderr + opslog pipeline, plus secret redaction on all sinks. Default-off: with no OTEL_* / SILO_OTEL_ENABLED config, behavior is unchanged. Bootstrap (internal/telemetry): - Setup() builds one shared resource, a TracerProvider (parent-based trace-id ratio sampler), a LoggerProvider, and the W3C TraceContext+Baggage propagator from env. It installs NO MeterProvider — metrics stay on Prometheus, and the built-in no-op global MeterProvider keeps the trace instrumentation libs from double-emitting. Shutdown is deferred with a flush timeout. - Logs are bridged via otelslog fan-out (slog.MultiHandler), level-gated by the shared LevelVar and best-effort so a failing collector can't break the console or DB branches. stderr + opslog stay untouched. Secret redaction (internal/logredact): - A slog.Handler masks secret-keyed attributes (password, token, api_key, authorization, cookie, ...) — including .With-bound attrs, nested groups, secret-keyed group subtrees, and values behind a LogValuer — on the console and OTLP sinks, with a no-op fast path when a record has no secret keys. opslog.shouldRedact delegates to logredact.SecretKey so all sinks share one marker list. Rotation is infra-managed (no custom file sink): container runtime for stderr, collector/backend for OTLP, opslog partition-pruning for the DB. Documented in docs/architecture/observability.md. Verification: go build ./..., go vet, gofmt -l — clean; go test ./internal/telemetry/ ./internal/logredact/ -race pass. AI-use disclosure: implemented with AI assistance (Claude Code), including adversarial reviews that hardened the bootstrap and fixed two redaction leak paths; reviewed by the author. * refactor(observability): slog context+component sweep, sloglint gate (phase 3) Part of #265. Builds on the OTel bootstrap + redaction commit. Standardizes every log call site onto the context-carrying slog variants so records correlate with the active OpenTelemetry trace, and locks the standard in with a machine gate so future code (human- or AI-authored) can't drift back. - Call-site sweep: converted the remaining slog.<Level>(...) calls to the slog.<Level>Context(ctx, ...) form wherever a context.Context is in scope (background/init calls with no ctx are left as-is), across 183 files. Applied via a type-aware AST codemod. Log levels and message strings are preserved verbatim; a component attr (canonical per-package name) is added to direct package-level slog calls. Bound-logger calls keep their existing .With bindings. The main.go and telemetry package conversions rode with their file in the previous commit to keep each file within a single commit. - Enforcement (.golangci.yml): enable sloglint with context=scope, static-msg, key-naming-case=snake, no-mixed-args. After the sweep all four report zero violations repo-wide (tests included), so make lint / CI now blocks any regression to the non-context form. The gate ships with the sweep because it cannot be green until the legacy sites are converted. Metrics remain on Prometheus; no behavior change to /metrics or Grafana. Verification: go build ./..., go vet ./..., gofmt -l — clean; sloglint (all 4 rules) 0 violations repo-wide; log levels verified unchanged. AI-use disclosure: implemented with AI assistance (Claude Code), including the codemod; reviewed by the author. * fix(observability): honor per-signal OTLP protocol and secret WithGroup names Two Codex review findings on PR #290: - telemetry: OTEL_EXPORTER_OTLP_{TRACES,LOGS}_PROTOCOL now override the generic OTEL_EXPORTER_OTLP_PROTOCOL per signal, so mixed collector setups (e.g. HTTP logs + gRPC traces) build the right exporter. - logredact: entering a group whose name is secret-bearing (e.g. WithGroup("authorization")) now masks every leaf in that subtree, matching how slog.Group("authorization", ...) is masked as a whole. * fix(observability): address review feedback on telemetry bootstrap - Telemetry setup failure no longer kills boot: Setup returns usable no-op providers alongside the error and main logs and continues with telemetry disabled, honoring the best-effort contract. - Honor OTEL_TRACES_SAMPLER (always_on/off, traceidratio, parentbased_* variants); unsupported values fall back to parentbased_traceidratio. - Attach node identity as semconv service.instance.id instead of the non-semconv node.name. - Rename opslog retention-scope log attrs to target_component/target_level so they no longer collide with the canonical component routing key, and tag those lines with component=opslog. - Fix stale levelGated comment casing; use WarnContext in the telemetry shutdown defer; document the LogValuer double-resolve on the redaction slow path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Quick <31828688+Quick104@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
417 lines
13 KiB
Go
417 lines
13 KiB
Go
package abs
|
|
|
|
import (
|
|
"encoding/json"
|
|
"errors"
|
|
"io"
|
|
"log/slog"
|
|
"net/http"
|
|
|
|
"github.com/oklog/ulid/v2"
|
|
|
|
"github.com/Silo-Server/silo-server/internal/audiobooks/smartcoll"
|
|
"github.com/Silo-Server/silo-server/internal/models"
|
|
)
|
|
|
|
// smartCollectionBody is the JSON body for POST and PATCH
|
|
// /me/smart-collections[/{id}]. Pointer fields support partial PATCH.
|
|
type smartCollectionBody struct {
|
|
Name *string `json:"name"`
|
|
Description *string `json:"description"`
|
|
Color *string `json:"color"`
|
|
IsPublic *bool `json:"isPublic"`
|
|
IsPinned *bool `json:"isPinned"`
|
|
QueryDef *smartcoll.QueryDefinition `json:"query_def"`
|
|
}
|
|
|
|
// handleCreateSmartCollection — POST /me/smart-collections.
|
|
func (h *Handler) handleCreateSmartCollection(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
http.Error(w, "smart collection store unavailable", http.StatusServiceUnavailable)
|
|
return
|
|
}
|
|
|
|
var body smartCollectionBody
|
|
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil {
|
|
http.Error(w, "invalid body", http.StatusBadRequest)
|
|
return
|
|
}
|
|
if body.Name == nil || *body.Name == "" {
|
|
http.Error(w, "name required", http.StatusBadRequest)
|
|
return
|
|
}
|
|
|
|
c := SmartCollection{
|
|
ID: ulid.Make().String(),
|
|
UserID: a.UserID,
|
|
ProfileID: a.ProfileID,
|
|
Name: *body.Name,
|
|
}
|
|
if body.Description != nil {
|
|
c.Description = *body.Description
|
|
}
|
|
if body.Color != nil {
|
|
c.Color = *body.Color
|
|
}
|
|
if body.IsPublic != nil {
|
|
c.IsPublic = *body.IsPublic
|
|
}
|
|
if body.IsPinned != nil {
|
|
c.IsPinned = *body.IsPinned
|
|
}
|
|
|
|
qd := smartcoll.QueryDefinition{}
|
|
if body.QueryDef != nil {
|
|
qd = *body.QueryDef
|
|
}
|
|
qd = qd.Normalize()
|
|
if err := qd.Validate(true); err != nil {
|
|
http.Error(w, "invalid query_def: "+err.Error(), http.StatusBadRequest)
|
|
return
|
|
}
|
|
qdBytes, err := json.Marshal(qd)
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection marshal query_def failed", "component", "audiobooks", "err", err)
|
|
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
c.QueryDef = qdBytes
|
|
|
|
if err := h.deps.SmartCollectionStore.CreateSmartCollection(r.Context(), c); err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection create failed", "component", "audiobooks", "err", err, "user", a.UserID)
|
|
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
|
|
persisted, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), c.ID)
|
|
if errors.Is(err, ErrNotFound) || err != nil {
|
|
persisted = c
|
|
}
|
|
writeJSON(w, http.StatusOK, smartCollectionToABS(persisted))
|
|
}
|
|
|
|
func (h *Handler) handleListSmartCollections(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
writeJSON(w, http.StatusOK, map[string]any{"items": []any{}})
|
|
return
|
|
}
|
|
rows, err := h.deps.SmartCollectionStore.ListUserSmartCollections(r.Context(), a.UserID, a.ProfileID)
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection list failed", "component", "audiobooks", "err", err, "user", a.UserID)
|
|
http.Error(w, "smart collection list failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
out := make([]map[string]any, 0, len(rows))
|
|
for _, c := range rows {
|
|
out = append(out, smartCollectionToABS(c))
|
|
}
|
|
writeJSON(w, http.StatusOK, map[string]any{"items": out})
|
|
}
|
|
|
|
func (h *Handler) handleGetSmartCollection(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), chiURLID(r))
|
|
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID) && !c.IsPublic) {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection get failed", "component", "audiobooks", "err", err)
|
|
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
writeJSON(w, http.StatusOK, smartCollectionToABS(c))
|
|
}
|
|
|
|
func (h *Handler) handleUpdateSmartCollection(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
id := chiURLID(r)
|
|
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
|
|
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID)) {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection get-for-update failed", "component", "audiobooks", "err", err, "id", id)
|
|
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
|
|
var body smartCollectionBody
|
|
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&body); err != nil {
|
|
http.Error(w, "invalid body", http.StatusBadRequest)
|
|
return
|
|
}
|
|
if body.Name != nil {
|
|
c.Name = *body.Name
|
|
}
|
|
if body.Description != nil {
|
|
c.Description = *body.Description
|
|
}
|
|
if body.Color != nil {
|
|
c.Color = *body.Color
|
|
}
|
|
if body.IsPublic != nil {
|
|
c.IsPublic = *body.IsPublic
|
|
}
|
|
if body.IsPinned != nil {
|
|
c.IsPinned = *body.IsPinned
|
|
}
|
|
if body.QueryDef != nil {
|
|
qd := body.QueryDef.Normalize()
|
|
if err := qd.Validate(true); err != nil {
|
|
http.Error(w, "invalid query_def: "+err.Error(), http.StatusBadRequest)
|
|
return
|
|
}
|
|
qdBytes, mErr := json.Marshal(qd)
|
|
if mErr != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection marshal query_def failed", "component", "audiobooks", "err", mErr)
|
|
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
c.QueryDef = qdBytes
|
|
}
|
|
if err := h.deps.SmartCollectionStore.UpdateSmartCollection(r.Context(), c); err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection update failed", "component", "audiobooks", "err", err, "id", id)
|
|
http.Error(w, "smart collection persist failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
persisted, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
|
|
if err != nil {
|
|
persisted = c
|
|
}
|
|
writeJSON(w, http.StatusOK, smartCollectionToABS(persisted))
|
|
}
|
|
|
|
func (h *Handler) handleDeleteSmartCollection(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
id := chiURLID(r)
|
|
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
|
|
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID)) {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection get-for-delete failed", "component", "audiobooks", "err", err, "id", id)
|
|
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
if err := h.deps.SmartCollectionStore.DeleteSmartCollection(r.Context(), id); err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection delete failed", "component", "audiobooks", "err", err, "id", id)
|
|
http.Error(w, "smart collection delete failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
w.WriteHeader(http.StatusNoContent)
|
|
}
|
|
|
|
// handleSmartCollectionItems — GET /me/smart-collections/{id}/items.
|
|
// Evaluates the collection's query_def against the audiobook catalog
|
|
// and returns a paged envelope. When the caller is the owner, per-user
|
|
// state is hydrated; non-owner viewing a public collection sees
|
|
// personalized rules silently dropped.
|
|
func (h *Handler) handleSmartCollectionItems(w http.ResponseWriter, r *http.Request) {
|
|
a, ok := absAuthFrom(r)
|
|
if !ok || a.UserID == "" {
|
|
http.Error(w, "unauthorized", http.StatusUnauthorized)
|
|
return
|
|
}
|
|
if h.deps.SmartCollectionStore == nil {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
id := chiURLID(r)
|
|
c, err := h.deps.SmartCollectionStore.GetSmartCollection(r.Context(), id)
|
|
if errors.Is(err, ErrNotFound) || (err == nil && !sameABSPrincipal(a, c.UserID, c.ProfileID) && !c.IsPublic) {
|
|
http.Error(w, "smart collection not found", http.StatusNotFound)
|
|
return
|
|
}
|
|
if err != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection items get failed", "component", "audiobooks", "err", err, "id", id)
|
|
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
|
|
var qd smartcoll.QueryDefinition
|
|
if len(c.QueryDef) > 0 {
|
|
if uErr := json.Unmarshal(c.QueryDef, &qd); uErr != nil {
|
|
slog.ErrorContext(r.Context(), "abs smart collection invalid stored query_def", "component", "audiobooks", "err", uErr, "id", id)
|
|
http.Error(w, "smart collection get failed", http.StatusInternalServerError)
|
|
return
|
|
}
|
|
}
|
|
qd = qd.Normalize()
|
|
|
|
limit, page := readPagedQuery(r, 30)
|
|
if r.URL.Query().Get("limit") == "" && qd.Limit != nil && *qd.Limit > 0 {
|
|
limit = *qd.Limit
|
|
}
|
|
|
|
access, err := h.accessFilterForAuth(r.Context(), a)
|
|
if err != nil {
|
|
http.Error(w, "resolve access: "+err.Error(), http.StatusForbidden)
|
|
return
|
|
}
|
|
allLibs, err := h.deps.MediaStore.ListAudiobookLibraries(r.Context(), access)
|
|
if err != nil {
|
|
slog.WarnContext(r.Context(), "abs smart collection libraries fetch failed", "component", "audiobooks", "err", err, "id", id)
|
|
allLibs = nil
|
|
}
|
|
libByID := make(map[int64]AudiobookLibrary, len(allLibs))
|
|
for _, lib := range allLibs {
|
|
libByID[lib.ID] = lib
|
|
}
|
|
var targetLibs []AudiobookLibrary
|
|
if len(qd.LibraryIDs) > 0 {
|
|
for _, lid := range qd.LibraryIDs {
|
|
if lib, ok := libByID[lid]; ok {
|
|
targetLibs = append(targetLibs, lib)
|
|
}
|
|
}
|
|
} else {
|
|
targetLibs = allLibs
|
|
}
|
|
|
|
owner := sameABSPrincipal(a, c.UserID, c.ProfileID)
|
|
progressByID := map[string]ProgressRow{}
|
|
bookmarkCountByID := map[string]int{}
|
|
if owner {
|
|
if h.deps.ProgressStore != nil {
|
|
if rows, perr := h.deps.ProgressStore.ListProgressForAudiobooks(r.Context(), a.UserID, a.ProfileID, 10000); perr == nil {
|
|
for _, p := range rows {
|
|
progressByID[p.ContentID] = p
|
|
}
|
|
}
|
|
}
|
|
if h.deps.BookmarkStore != nil {
|
|
if counts, berr := h.deps.BookmarkStore.CountByUser(r.Context(), a.UserID, a.ProfileID); berr == nil {
|
|
bookmarkCountByID = counts
|
|
}
|
|
}
|
|
}
|
|
|
|
candidates := make([]smartcoll.Candidate, 0, 256)
|
|
for _, lib := range targetLibs {
|
|
items, _, lerr := h.deps.MediaStore.ListAudiobooks(r.Context(), lib.ID, 0, 0, access, Filter{})
|
|
if lerr != nil {
|
|
slog.WarnContext(r.Context(), "abs smart collection list-audiobooks failed", "component", "audiobooks", "err", lerr, "library", lib.ID)
|
|
continue
|
|
}
|
|
for _, mi := range items {
|
|
cand := smartcoll.Candidate{Item: siloItemToSmartcollItem(mi)}
|
|
if owner {
|
|
if p, ok := progressByID[mi.ContentID]; ok {
|
|
cand.IsFinished = p.IsFinished
|
|
cand.ProgressPct = float32(p.ProgressPct)
|
|
cand.CurrentSeconds = int(p.CurrentSeconds)
|
|
cand.LastPlayedAt = p.UpdatedAt
|
|
}
|
|
cand.BookmarkCount = bookmarkCountByID[mi.ContentID]
|
|
}
|
|
candidates = append(candidates, cand)
|
|
}
|
|
}
|
|
|
|
matched := smartcoll.Evaluate(r.Context(), qd, candidates, smartcoll.EvaluateOptions{
|
|
AllowPersonalized: owner,
|
|
UserSeed: a.UserID + ":" + c.ID,
|
|
})
|
|
|
|
total := len(matched)
|
|
start := page * limit
|
|
if start > total {
|
|
start = total
|
|
}
|
|
end := start + limit
|
|
if end > total {
|
|
end = total
|
|
}
|
|
pageSlice := matched[start:end]
|
|
|
|
libDefault := h.resolveDefaultLibrary(r.Context(), access)
|
|
libDefaultID := audiobookLibraryID(libDefault)
|
|
results := make([]map[string]any, 0, len(pageSlice))
|
|
for _, cand := range pageSlice {
|
|
entry := map[string]any{
|
|
"id": cand.Item.ID,
|
|
"libraryId": libDefaultID,
|
|
"media": map[string]any{
|
|
"metadata": map[string]any{"title": cand.Item.Title},
|
|
},
|
|
}
|
|
results = append(results, entry)
|
|
}
|
|
|
|
writeJSON(w, http.StatusOK, pagedEnvelope(results, total, limit, page, qd.Sort.Field, qd.Sort.Order == "desc", "", false, ""))
|
|
}
|
|
|
|
// siloItemToSmartcollItem maps a silo *models.MediaItem into the
|
|
// audiobook-domain Item shape the smartcoll evaluator walks.
|
|
func siloItemToSmartcollItem(mi *models.MediaItem) smartcoll.Item {
|
|
if mi == nil {
|
|
return smartcoll.Item{}
|
|
}
|
|
it := smartcoll.Item{
|
|
ID: mi.ContentID,
|
|
Title: mi.Title,
|
|
Genres: mi.Genres,
|
|
Year: mi.Year,
|
|
Language: mi.OriginalLanguage,
|
|
DurationSeconds: mi.Runtime,
|
|
}
|
|
for _, p := range mi.People {
|
|
switch p.Kind {
|
|
case models.PersonKindAuthor:
|
|
it.Authors = append(it.Authors, p.Name)
|
|
case models.PersonKindNarrator:
|
|
it.Narrators = append(it.Narrators, p.Name)
|
|
}
|
|
}
|
|
for _, s := range mi.AudiobookSeries {
|
|
it.Series = append(it.Series, s.Name)
|
|
}
|
|
if len(mi.Studios) > 0 {
|
|
it.Publisher = mi.Studios[0]
|
|
}
|
|
if mi.RatingIMDB != nil {
|
|
it.Rating = *mi.RatingIMDB
|
|
}
|
|
if mi.AddedAt != nil {
|
|
it.AddedAt = *mi.AddedAt
|
|
}
|
|
return it
|
|
}
|