silo-server/docs/superpowers/plans/notifications/relay/01-implementation-plan.md

Silo Push Relay — Implementation Plan

Date: 2026-06-13 Status: Draft (implementation-grade, phased) Service: silo-push-relay (provisional repo name; the relay code lives in a separate repository) Provisional hostname: relay.silo.app Stack: Go 1.25+, PostgreSQL, Redis

Source documents (read before executing any phase):

• ./00-relay-spec.md — the engineering spec this plan implements. Section numbers below (e.g. "spec §6.4") refer to it.
• ./02-apns-fcm-2026-reference.md — 2026-current Apple/Google/Go reference (cited as "reference §N").
• ../02-apns-relay.md and ../03-fcm-relay.md — the authoritative external wire contract (the 02/03 request/response shapes). The relay must honor these byte-for-byte.

Commands assume the repository root is the cwd. This document contains no local absolute filesystem paths or transient worktree IDs; all repo references are repository-relative.

Repo boundary. This plan describes a separate repository (silo-push-relay). The design docs are authored here in silo-server purely for co-location with the 02/03 contracts. The relay code lives elsewhere. The one deliberate coupling is the shared pushwire payload-builder package, mirrored into both repos (see §3 and Phase 3/4).

---

1. Overview

1.1 What is being built

A small, stateless-on-the-request-path Go HTTP service that holds the official Silo Apple (.p8 APNs auth key) and Google (Firebase service-account JSON) push credentials, accepts authenticated content-free push requests from opted-in self-hosted Silo servers on two endpoints, builds a fixed generic APNs/FCM payload, forwards it upstream, and maps the upstream result back to a narrow caller-facing response.

The two send endpoints and their contract are fixed by 02/03 and reproduced in spec §5:

Endpoint	Purpose	Authoritative contract
POST /v1/apple/send	Forward one opaque push to APNs for one device token	../02-apns-relay.md, spec §5.1
POST /v1/fcm/send	Forward one opaque data-only push to FCM for one device token	../03-fcm-relay.md, spec §5.2
GET /healthz	Liveness (public)	spec §5.3
GET /readyz	Readiness incl. PG/Redis core deps + per-provider credential health (internal)	spec §5.4
GET /metrics	Prometheus exposition (internal)	endpoint spec §5.4; metric catalog §13.2

Administration is a CLI (relayctl) that writes directly to the relay's PostgreSQL — there is no public admin HTTP API in v1 (spec §5.6).

1.2 The v1 cut line

In scope (v1):

• POST /v1/apple/send and POST /v1/fcm/send exactly per 02/03.
• Bearer auth (rk_ keys), per-account allowlists, Redis token-bucket rate limiting, Redis idempotency.
• APNs token-based (ES256 .p8) client with cached JWT; FCM HTTP v1 client with cached OAuth2 token.
• PostgreSQL storage of accounts/keys/allowlists/redacted op-logs only; Goose-style migrations.
• relayctl admin CLI; structured redacted logs; Prometheus metrics; /healthz + /readyz; graceful shutdown.
• Single region, multiple stateless replicas behind a TLS load balancer.

Explicitly out of scope (v1) — from spec §2.2 / NG1–NG7:

• Token→user aliases, device subscriptions, any user/profile identity (stateless v1).
• Arbitrary APNs/FCM JSON, custom titles/bodies, topic/condition broadcasts, badge-by-default.
• Delivery receipts, analytics, open-tracking.
• APNs broadcast / Live Activity channels (/4/broadcasts/...).
• Web Push, HMS, ADM, or any non-APNs/non-FCM transport.
• A public self-service signup / billing surface.
• The custom_apns / custom_fcm direct paths (those run inside silo-server).
• Multi-region (the request path is stateless, but Redis/Postgres regional strategy is deferred).

1.3 The real long pole: provider account provisioning

Apple/Firebase account provisioning is the critical-path dependency, not the code. Obtaining the official Apple Developer team, generating a .p8 APNs auth key (Key ID + Team ID), registering per-platform bundle topics, standing up the Firebase project(s), and minting a service-account JSON can take days-to-weeks of organizational/approval lead time and is fully decoupled from coding.

Action: start the §7 provisioning checklist on day 1, in parallel with Phase 0. Phases 0–2 (scaffold, storage, auth/rate-limit/idempotency) need no live provider credentials and can complete entirely against fakes and contract tests. Only Phase 3 (APNs integration tests against sandbox) and Phase 4 (FCM validateOnly) are gated on credentials. If provisioning slips, the code still reaches "everything but live upstream" without blocking.

flowchart LR
  subgraph Track A — Code
    P0[Phase 0\nscaffold] --> P1[Phase 1\nstorage] --> P2[Phase 2\nauth/RL/idem]
    P2 --> P3[Phase 3\nAPNs]
    P2 --> P4[Phase 4\nFCM]
    P3 --> P5[Phase 5\nobservability]
    P4 --> P5
    P5 --> P6[Phase 6\nharden+deploy]
  end
  subgraph Track B — Provisioning (parallel, long pole)
    A1[Apple team + .p8 + topics]
    G1[Firebase project + SA JSON + package]
  end
  A1 -. gates .-> P3
  G1 -. gates .-> P4
  A1 -. gates .-> P6
  G1 -. gates .-> P6

---

2. Repository Bootstrap

2.1 Go module path

module github.com/silo-app/silo-push-relay

Target Go 1.25+ (required by jackc/pgx v5.10 and firebase.google.com/go/v4 v4.20; reference §3 version-targeting note). Pin in go.mod with go 1.25.

2.2 Directory layout

silo-push-relay/
├── cmd/
│   ├── relay/                  # main HTTP service entrypoint (wires config → stores → clients → server)
│   │   └── main.go
│   └── relayctl/               # admin CLI (account/key/allowlist/logs/ping-upstream); writes directly to PG
│       └── main.go
├── internal/
│   ├── config/                 # env + secret-manager config loading, validation, defaults
│   ├── httpapi/                # router, handlers, middleware (auth, rate limit, idempotency, redaction), error bodies
│   ├── apns/                   # APNs upstream client: ES256 JWT lifecycle, HTTP/2 pool, reason→error mapping
│   ├── fcm/                    # FCM upstream client: OAuth2 token source, HTTP/2 client, status→error mapping
│   ├── accounts/              # data-access for accounts/keys/allowlists (shared by httpapi + relayctl)
│   ├── ratelimit/              # Redis Lua token-bucket limiter (per-account + coarse per-token)
│   ├── idempotency/            # Redis SET NX lock / replay / 409 / 422 store
│   ├── pushwire/               # SHARED payload builders (APNs + FCM payload/header construction) — mirrored into silo-server
│   ├── oplog/                  # redacted op-log writer (PG) + token hashing helper
│   ├── observability/          # slog JSON handler + redaction, Prometheus registry/metrics, /healthz /readyz /metrics
│   └── store/                  # pgxpool + go-redis bootstrapping, health pings, migration runner hook
├── migrations/
│   └── sql/                    # Goose-style timestamped SQL migrations (see §2.6)
├── deploy/
│   ├── Dockerfile
│   └── (k8s / compose manifests as applicable)
├── Makefile
├── go.mod
├── go.sum
├── .golangci.yml
├── .github/workflows/ci.yml    # or equivalent CI config
└── README.md

Package ownership mirrors the Silo convention (CLAUDE.md "keep new code in the package that owns the behavior"): no catch-all utils package; shared payload logic lives in pushwire; shared data-access lives in accounts/store so relayctl and httpapi share schema and constraints (spec §5.6 "The CLI shares the relay's data-access package").

2.3 The shared pushwire package (maintainability win)

internal/pushwire is the single source of truth for how an opaque request becomes an upstream payload + headers — the APNs aps/silo dictionaries and apns-* headers (spec §5.1), and the FCM data-only message body + android config (spec §5.2). It is intentionally dependency-light (pure Go structs + JSON, no network) so it can be mirrored into silo-server under its custom_apns / custom_fcm paths, where the self-hosted server builds the same payloads when an admin uses their own credentials. One package, two repos: the relay and the direct-credential path can never drift on payload shape. (See §3 dependency table and Phase 3/4.)

Mirroring mechanism & sync runbook (drift/ownership hazard — make it explicit). Keep pushwire import-cycle-free and free of relay-only types (no DB, no config). Designate one repo as the canonical source of truth for pushwire (the relay repo silo-push-relay, since it owns the upstream payload contract). The copy in silo-server is a checked-in generated artifact, not a hand-edited file: commit a content checksum of the canonical package alongside it, and have both repos' CI verify their local copy against that checksum (a copied package in repo A cannot fail repo B's CI on its own, so a passive golden-file test in one repo is insufficient). Bump a versioned payload-schema constant on any payload change so a mismatch is unambiguous. Add a golden-file + closed-allowlist-key contract test on both sides (Phase 3/4 / §5.2). Document the ownership/sync runbook (who regenerates, in what order, how the checksum is bumped) so a payload change cannot silently diverge until the other repo happens to update. The preferred end state is a small shared module both repos import rather than a manual copy.

2.4 Config strategy

Twelve-factor-ish, but credentials come from a secret manager, not env vars (spec §11, reference §4.5 ranking: secret-manager > mounted file > env var > in-image). internal/config resolves, in order of preference per secret:

Config	Source (preferred → fallback)	Notes
APNs .p8 private key, Key ID, Team ID	secret manager → mounted file	never env var, never in image
FCM SA JSON or Workload Identity binding	GCP attached SA / Workload Identity → secret manager JSON	off-GCP uses JSON; on-GCP keyless (reference §4.5)
API-key HMAC pepper	secret manager	rotation is heavier (spec open Q#6)
Postgres DSN, Redis URL	secret manager → env	pool-tuned
Non-secret tuning (timeouts, rate defaults, listen addrs, log level, environment label)	env vars / flags	safe to log

config.Load(ctx) returns a validated, fully-populated struct or a hard error; the process refuses to start with a missing/invalid credential. Provide a config.Validate() that cmd/relay calls before opening any listener. Non-secret defaults (rate limits from spec §9.2, timeouts from spec §11) live as constants with env overrides.

2.5 Makefile (mirror Silo's Makefile-driven workflow)

make targets, modeled on silo-server's Makefile (CLAUDE.md §Build):

make build            # go build ./cmd/relay ./cmd/relayctl
make run              # run cmd/relay locally against docker-compose PG+Redis
make test             # go test ./... (unit + contract; no live providers)
make test-integration # go test -tags=integration ./...  (APNs sandbox + FCM validateOnly; needs creds)
make lint             # golangci-lint run
make fmt              # gofmt -w + goimports
make migrate-create NAME=add_thing   # timestamped Goose migration scaffold
make migrate-up       # apply migrations
make migrate-status   # list migration state
make relayctl ARGS="account list"    # build+run the admin CLI
make docker           # build deploy/Dockerfile
make loadtest         # run the k6/vegeta burst scenario (Phase 6)

make migrate-create NAME=... must produce a timestamped filename (CLAUDE.md: "New migrations must use timestamped filenames created with make migrate-create; do not run goose fix or create paired .up.sql/.down.sql files"). The relay uses single-file Goose SQL migrations with -- +goose Up / -- +goose Down.

2.6 Migrations (Goose-style)

migrations/sql/<timestamp>_<name>.sql, run by a Goose runner wired in internal/store and invokable via make migrate-up and on cmd/relay startup (behind a flag). The v1 schema is exactly spec §8.1 (relay_accounts, relay_api_keys, relay_apns_allowlist, relay_fcm_allowlist, relay_op_logs). IDs are ULIDs stored as text; timestamps are timestamptz (Silo convention).

2.7 CI outline

CI (GitHub Actions or equivalent) runs on every push/PR:

jobs:
  build-test:
    steps:
      - setup-go 1.25
      - make lint                # golangci-lint
      - make build
      - make test                # unit + contract tests (no provider creds)
      - upload coverage
  migrate-check:
    services: [postgres, redis]
    steps:
      - make migrate-up          # migrations apply cleanly on a fresh DB
      - make migrate-status
  integration:                   # gated: only when provider creds are available (Phase 3+)
    if: secrets.APNS_P8 != '' && secrets.FCM_SA_JSON != ''
    services: [postgres, redis]
    steps:
      - make test-integration    # APNs sandbox + FCM validateOnly contract tests

The integration job is conditional on secrets being present so Phases 0–2 are green long before any credential exists.

---

3. Dependency Choices

All versions are from the reference §3.7 pinned list. Pin exact versions in go.mod; bump deliberately.

Concern	Library	Version	One-line rationale
APNs client	github.com/sideshow/apns2 or hand-rolled	v0.25.0 / x/net/http2 v0.56.0 + golang-jwt/jwt/v5 v5.3.1	De-facto Go APNs HTTP/2 client (auto JWT + conn reuse); decide vs. hand-rolled before Phase 3 because apns2 has had no release since Oct 2024 (bus-factor; reference §3.1, spec open Q#3).
FCM auth	golang.org/x/oauth2/google	v0.36.0	CredentialsFromJSONWithType(..., google.ServiceAccount, firebase.messaging scope) gives an auto-caching/-refreshing TokenSource; lighter than the Admin SDK and gives direct control of conn reuse + error classification (reference §3.2 — preferred for a fixed payload).
FCM (alt, batteries-included)	firebase.google.com/go/v4/messaging	v4.20.0	Acceptable alternative with IsUnregistered/IsQuotaExceeded helpers; pick exactly one FCM path, not both (reference §3.2). Plan defaults to the x/oauth2 path.
JWT (ES256 sign for APNs; RS256 for hand-rolled FCM grant)	github.com/golang-jwt/jwt/v5	v5.3.1	Maintained JWT lib; ES256 for APNs, RS256 if hand-rolling the FCM token exchange (reference §3.1).
HTTP/2 transport tuning	golang.org/x/net/http2	v0.56.0	ReadIdleTimeout+PingTimeout to reap half-open upstream conns — "the single most important tuning for a long-lived relay" (reference §3.3).
Postgres	github.com/jackc/pgx/v5 + pgxpool	v5.10.0	Recommended driver/pool; pgxpool.New is concurrency-safe + health-checked (reference §3.4).
Redis	github.com/redis/go-redis/v9	v9.20.1	Context-first client with pooling; atomic Lua for rate-limit, SET NX for idempotency (reference §3.4).
Router	stdlib net/http (Go 1.22+ http.ServeMux patterns)	stdlib	Two routes + health/metrics need no framework; stdlib ServeMux supports method+path patterns and gets HTTP/2 over TLS automatically. Avoids a dependency; middleware is plain http.Handler wrapping.
Structured logging	log/slog	stdlib (Go 1.21+)	slog.NewJSONHandler for production; redaction middleware so secrets can't leak (reference §3.5, spec §13.1).
Metrics	github.com/prometheus/client_golang	v1.23.2	Custom registry + promhttp.HandlerFor for /metrics; per-provider latency histograms + outcome counters (reference §3.5, spec §13.2).
Per-instance rate (outbound pacing helper)	golang.org/x/time/rate	current	Reservation.Delay() to compute relative Retry-After; used for in-process pacing, not the cross-replica cap (that is Redis — reference §4.2).
ULID generation	github.com/oklog/ulid/v2 (or equivalent)	current	Generate request_id and entity IDs as ULIDs (Silo convention; spec §8.1).

Decision to make before Phase 3 (spec open Q#3): adopt sideshow/apns2 or hand-roll the APNs client. Recommendation: prototype both in Phase 3 behind the internal/apns interface; pick apns2 if its recent commit/issue activity is acceptable, else hand-roll on x/net/http2 + golang-jwt/jwt/v5 mirroring the verified apns2 internals (reference §3.1: HostProduction/HostDevelopment, TokenTimeout=3000s, ReadIdleTimeout=15s). Either way the rest of the service depends only on the internal/apns interface, not the concrete client.

pushwire maintainability note (reference + spec §5). The shared pushwire package builds the exact APNs/FCM payloads and headers. Because silo-server's custom_apns/custom_fcm paths must produce identical payloads, pushwire is mirrored into silo-server with one canonical source repo (the relay), a committed checksum verified in CI on both sides, and a closed-allowlist-key test so an added data/payload key fails CI regardless of golden-file regeneration order (§2.3 sync runbook, §5.2). This is also a supply-chain control for the content-free guarantee: the payload-builder that enforces "data-only, no notification block, fixed key set" is enforced in two repos that must not drift. Payload shape is defined once.

---

4. Phased Tasks

Effort sizes: S ≈ 1 day, M ≈ 2–3 days, L ≈ 4–5 days (one engineer). Sizes are rough.

Phase 0 — Scaffold, config, health, CI (size: M)

Goal. A buildable, lintable, testable repo that boots cmd/relay, serves /healthz, loads config, and is green in CI — with no provider dependency.

Files to create/modify.

• go.mod, go.sum, .golangci.yml, Makefile, README.md, .github/workflows/ci.yml, deploy/Dockerfile.
• cmd/relay/main.go — load config, build slog JSON logger, start an http.Server with explicit timeouts and graceful shutdown, mount /healthz.
• internal/config/config.go — Load(ctx), Validate(), defaults; secret-manager interface stub (real wiring in Phase 6).
• internal/observability/logger.go — slog.NewJSONHandler setup.
• internal/httpapi/server.go, internal/httpapi/router.go, internal/httpapi/health.go — router, /healthz.
• internal/httpapi/errors.go — the standard error body (spec §5.5) and a writeError(w, status, code, msg, requestID) helper.

Implementation notes.

• Server timeouts are mandatory (zero = no timeout): ReadHeaderTimeout, ReadTimeout, WriteTimeout, IdleTimeout, MaxHeaderBytes (reference §3.3, spec §11). Body size cap ~16 KiB.
• Graceful shutdown: on SIGTERM/SIGINT call srv.Shutdown(ctx) with a bounded context (spec §14.2).
• /healthz returns 200 {"status":"ok"}, unauthenticated, no dependency checks (spec §5.3).
• Error body shape is fixed (spec §5.5):

  { "error": { "code": "string_code", "message": "human readable", "request_id": "01JRELAY..." } }
  

  Every request gets a ULID request_id, echoed as the X-Request-Id response header.

Tests.

• Unit: config.Validate rejects missing required fields; writeError emits exact JSON + X-Request-Id.
• Unit: /healthz returns 200 with the exact body.
• CI: make lint, make build, make test green; migrate-check job present (no migrations yet → no-op pass).

Acceptance criteria.

• make build produces relay and relayctl binaries.
• make lint and make test pass in CI with zero provider credentials.
• cmd/relay boots, serves /healthz 200, and shuts down gracefully on SIGTERM.
• Every response carries X-Request-Id; errors use the §5.5 shape.

---

Phase 1 — Storage, migrations, relayctl admin CLI (size: L)

Goal. The full PostgreSQL schema, a migration runner, the shared data-access package, and a working relayctl that can provision accounts/keys/allowlists and read op-logs — all without touching providers.

Files to create/modify.

• migrations/sql/<ts>_init_relay_schema.sql — exact schema from spec §8.1: relay_accounts, relay_api_keys, relay_apns_allowlist, relay_fcm_allowlist, relay_op_logs, plus indexes (relay_api_keys_prefix_uidx, relay_api_keys_account_idx, relay_op_logs_account_time_idx, relay_op_logs_time_idx).
• internal/store/postgres.go — pgxpool.New(ctx, dsn), Ping, migration runner hook.
• internal/store/redis.go — go-redis client + PING (used Phase 2).
• internal/accounts/ — data-access: CreateAccount, ListAccounts, DisableAccount, IssueKey, ListKeys, RevokeKey, LookupKeyByPrefix, SetApnsAllowlist, SetFcmAllowlist, GetAllowlists. Used by both httpapi and relayctl.
• internal/accounts/keys.go — key generation + hashing (see notes).
• internal/oplog/oplog.go — Write(ctx, entry) to relay_op_logs; TokenHash(token) string (SHA-256 hex).
• cmd/relayctl/main.go + subcommands per spec §5.6 table.

Implementation notes.

• API-key format & hashing (spec §8.2, reference §4.1):
  • Format rk_<env>_<base62/base64url>; env ∈ {live,test}. Random secret = 32 bytes from crypto/rand.
  • Prefix = rk_<env>_<first 6–8 chars of random>, non-secret, stored cleartext, uniquely indexed.
  • Store key_hash = HMAC-SHA256(pepper, secret) (32 bytes). Never store plaintext. Fast keyed hash is correct for 256-bit random keys (do not use argon2id here).
  • relayctl key issue prints the full rk_… exactly once to stdout; only hash+prefix persist (spec §8.3).
• relayctl connects directly to Postgres with a privileged DSN held by operators (no public admin API; spec §5.6). Each mutating command writes an admin.* op-log with operator identity (OS user / --actor) and never the secret token (spec §5.6).
• Allowlist commands replace the set atomically (UPSERT/DELETE in one tx) for relay_apns_allowlist / relay_fcm_allowlist.
• Official allowlist config values to seed in examples/tests (allowlist config, not the contract — spec §8.4): APNs topics com.continuum.app.ios, com.continuum.app.tvos, com.continuum.app.macos; FCM project continuum-prod-android, package com.continuum.app.android.

relayctl command surface (spec §5.6 / §13.3).

Command	Effect
relayctl account create --name <label> [--note <text>]	INSERT relay_accounts; print account_id
relayctl account list [--active]	SELECT accounts + key count + last activity
relayctl account disable --account <id>	status='disabled'
relayctl key issue --account <id> [--env live|test] [--expires <dur>]	INSERT key; print rk_… once
relayctl key list --account <id>	SELECT (prefix/created/last_used/revoked; never secret)
relayctl key revoke --key-prefix <prefix>	revoked_at=now()
relayctl allowlist apns set --account <id> --topics a,b,c	replace APNs allowlist
relayctl allowlist fcm set --account <id> --project <p> --packages a,b	replace FCM allowlist
relayctl allowlist show --account <id>	print effective allowlists
relayctl logs --account <id> [--since …] [--outcome rejected]	query relay_op_logs (redacted)
relayctl logs --gc	retention GC: delete success rows >14d, failure/reject rows >90d (Phase 5; cron-schedulable)
relayctl ping-upstream --provider apns|fcm --env sandbox	credential-only health check; flips relay_credential_healthy (Phase 3/4/6)

Tests.

• Integration (PG via docker-compose): make migrate-up applies cleanly on a fresh DB; make migrate-status lists state.
• Unit/integration: IssueKey → LookupKeyByPrefix round-trips; hash is HMAC-SHA256; plaintext never stored.
• Integration: full relayctl flow — create account → issue key → set allowlists → list/show; admin.* op-logs written without secrets.
• Unit: oplog.TokenHash is stable SHA-256 hex; op-log entry rejects/strips raw token.

Acceptance criteria.

• Migrations apply on a fresh DB and match spec §8.1 exactly (column names/types/constraints/indexes).
• relayctl can provision an account, issue a key (printed once), set both allowlists, and read redacted logs.
• No table stores tokens, content, or identity; relay_op_logs holds only token_hash, never raw tokens.
• Key hash is HMAC-SHA256(pepper, secret); constant-time compare verified in a unit test.

---

Phase 2 — Auth middleware, rate limit, idempotency (size: L)

Goal. The full request-path middleware chain (auth → rate limit → idempotency → decode/validate/allowlist) wired in front of stub send handlers that return a deterministic fake upstream result. No real provider yet.

Files to create/modify.

• internal/httpapi/middleware_auth.go — bearer parse, prefix lookup, constant-time compare, revoked/expired/disabled rejection, throttled last_used_at, per-IP anti-brute-force.
• internal/ratelimit/limiter.go + ratelimit/bucket.lua — Redis Lua token bucket (per-account + coarse per-token).
• internal/httpapi/middleware_ratelimit.go — applies the limiter, emits 429 + Retry-After.
• internal/idempotency/store.go + idempotency/idem.lua (if needed) — SET NX lock, replay, 409, 422.
• internal/httpapi/middleware_idempotency.go — wraps send handlers.
• internal/httpapi/decode.go — DisallowUnknownFields decoder + field validation + allowlist enforcement.
• internal/httpapi/apple.go, internal/httpapi/fcm.go — handler skeletons returning a fake accepted result (real upstream in Phase 3/4).

Implementation notes.

• Auth (spec §8.2): parse Authorization: Bearer rk_<env>_<rest> (malformed → 401 unauthorized); derive key_prefix; SELECT … WHERE key_prefix=$1 (unique index); reject if no row / revoked / expired / account disabled (do not distinguish reasons to caller — all 401 unauthorized); subtle.ConstantTimeCompare(HMAC(pepper, presented), key_hash); throttle last_used_at to ≤1/min/key; rate-limit auth failures per resolved client IP (see client-IP resolution below).
• Constant-time auth path (no enumeration oracle). On a missing/revoked/expired row, still perform a dummy HMAC-SHA256 against a fixed decoy hash with subtle.ConstantTimeCompare before returning the uniform 401, so "unknown prefix" is not measurably faster than "known prefix, wrong secret." Keep the fine-grained failure reason (unknown_prefix vs hash_mismatch) in internal logs/metrics only — never in a response and never exposed to an untrusted /metrics scraper.
• Client-IP resolution / trusted proxy (security-critical). The relay sits behind a TLS load balancer, so Go's http.Request.RemoteAddr is the LB address, not the caller. Derive the client IP from RemoteAddr unless the immediate peer is in an allowlisted trusted-proxy CIDR list (a config value), in which case take the right-most untrusted hop from the LB-set forwarding header. Caller-supplied X-Forwarded-For/X-Real-IP headers are ignored — trusting them would let an attacker spread brute-force across unlimited fake IPs (defeating the per-IP throttle) and poison egress_ip in op-logs. Both the per-IP auth-failure limiter and the logged egress_ip consume this single resolved IP. Account for shared egress (CGNAT/VPN — multiple self-hosted servers behind one IP, anticipated in spec §3.4): make the brute-force lockout soft (added latency / Retry-After), not a hard block, so a noisy/compromised neighbor cannot lock out legitimate servers sharing the IP.
• Bounded fail-open on Redis outage. The per-account/daily rate limit is the primary cap on a stolen-key attacker's spend (and on shared downstream APNs/FCM quota). A naive fail-open removes that cap entirely during a Redis outage. Instead, when Redis is unreachable, fall back to a conservative in-process per-account limiter (x/time/rate) so each account is still capped to a small multiple of its normal rate; the allowlist (always enforced from Postgres) remains the hard boundary. A Redis outage pages ops; the cross-replica daily quota cannot be enforced during the outage (documented limitation).
• Rate limit (spec §9): Redis HASH rl:{account_id} = {tokens, last_refill_ts} with TTL, atomic Lua (refill→check→consume in one EVAL). Defaults (spec §9.2): per-account ~10 req/s burst, refill 10/s with burst capacity 20; per-account daily 50,000 req/day; coarse (account_id, token_hash) ~1 req/3s. Over limit → 429 rate_limited with relative Retry-After (delay-seconds, never an absolute epoch — reference §4.2). On Redis down: bounded fail-open (in-process per-account fallback limiter, above — not unlimited), log rate_limit.degraded, increment relay_rate_limit_degraded_total (spec §9.4).
  • Daily quota mechanics. Store the daily counter as a separate Redis counter rl:day:{account_id}:{yyyymmdd} with INCR + EXPIRE to the end of the window — it is not the per-second token bucket and not a Postgres column. Reset semantics: fixed UTC calendar day (yyyymmdd in UTC). During a Redis outage the daily counter, like the per-second bucket, falls back to the bounded in-process limiter and the precise cross-replica daily cap cannot be enforced (documented; pages ops). Emit relay_rate_limited_total{kind="daily"} and alert on accounts approaching/hitting the cap (a big-release burst can legitimately reach 50k/day; §6.4).
• Project-level FCM circuit breaker (multi-tenant fairness). The ~600k msgs/min FCM project quota (reference §2.7) is shared across all relay accounts, but the per-account limit is local to each account — one noisy account can exhaust the shared quota for everyone. Add a relay-side global per-project outbound limiter / circuit breaker that tracks aggregate FCM send rate against the project quota and sheds load with 503 upstream_throttled before hammering FCM when near quota (the relay is the only aggregation point that can honor reference §2.7's anti-spike / smooth-traffic guidance — avoid sending within a 2-minute window of :00/:15/:30/:45). Export relay_fcm_project_quota_headroom and alert (§6.4).
• Idempotency (spec §10): require Idempotency-Key header on sends; missing → 400 missing_idempotency_key. Length-limit ≤255 chars; the three-part <delivery_id>:<device_id>:<attempt> shape is opaque (not strictly parsed). Namespaced idem:{account_id}:{key}. SET … NX EX <lock_ttl> (lock_ttl ≈ 30s) in-flight lock:
  • NX success → process; on completion overwrite with serialized {status, body, upstream_id, canonical_hash} + ~24h TTL (success and error recorded).
  • NX fail + stored completed result → replay verbatim (same apns_id/fcm_message_name).
  • NX fail + still in-flight → 409 idempotency_conflict.
  • Same key + different request payload (compare stored canonical_hash) → 422 idempotency_key_reuse.
  • Fails closed: Redis unavailable → 503 upstream_unavailable (never risk a double-send; spec §10.2).
  • Lock-TTL invariant (no double-send / no clobber). The upstream APNs/FCM call (plus HTTP/2 setup, JWT/OAuth refresh, GOAWAY reconnect) can otherwise outlast lock_ttl, letting a concurrent retry's NX succeed and fire a second upstream send. Pin the upstream call with a hard context deadline strictly shorter than lock_ttl (≈10s upstream vs 30s lock) and refuse to send upstream if the handler's own deadline has already passed the lock TTL. On completion, use a compare-and-set that only overwrites the stored value if it is still this request's in-flight marker, so a lock stolen by a concurrent request cannot have its result clobbered.
  • Privacy invariant (no stored token). The canonical_hash is computed over the validated fields with the device token replaced by its already-computed token_hash (SHA-256); the stored idempotency value is exactly {status, body, upstream_id, canonical_hash} and holds no raw token — matching the op-log no-raw-token guarantee (NG1, spec §3.2/§12).
  • Drain interaction. On handler context-cancellation (e.g. a drain/shutdown that severs an in-flight send), release or downgrade the in-flight marker so an immediate caller retry does not get spurious 409 idempotency_conflict for the full lock_ttl (see Phase 6 drain ordering and §5.7 drain test).
• Decode/validate/allowlist (spec §5.1/§5.2/§8.4): DisallowUnknownFields (unknown key → 400 unexpected_field). Validate every field per the spec field tables (token shape, environment, mode, ID length caps, collapse-id ≤64 bytes for APNs). Allowlist check after auth, before payload build: APNs topic ∈ account's relay_apns_allowlist else 403 topic_not_allowed; FCM (project_id, package_name) ∈ relay_fcm_allowlist else 403 project_not_allowed / 403 package_not_allowed. Allowlist may be a short-TTL per-account-scoped in-memory cache invalidated on relayctl allowlist … set.

Middleware order (spec §4.1).

TLS → auth → rate-limit → decode(DisallowUnknownFields) → validate+allowlist → idempotency → [Phase 3/4 upstream] → map → store idem result → op-log + metrics

Tests.

• Unit: bearer parse edge cases; constant-time compare; revoked/expired/disabled all map to 401; last_used_at throttle.
• Unit: the auth path is constant-time for unknown-prefix vs wrong-secret (dummy HMAC + ConstantTimeCompare on a missing/revoked row) — no enumeration timing oracle; the fine-grained reason stays internal-only.
• Unit: trusted-proxy client-IP resolution — a forged caller X-Forwarded-For/X-Real-IP is ignored; only the LB-set header from an allowlisted trusted-proxy peer is honored; RemoteAddr is used otherwise.
• Integration (Redis): token bucket refill/consume math; 429 + correct relative Retry-After; daily counter (rl:day:{account}:{yyyymmdd}, fixed-UTC reset, EXPIRE) trips at 50k; bounded fail-open on Redis down (in-process per-account cap engaged, not unlimited).
• Integration (Redis): idempotency first/replay/409/422/missing-key matrix (spec §10.3); fail-closed 503 on Redis down; the stored value carries no raw token (canonical hash uses token_hash).
• Integration (Redis): Redis dropped mid-request (not just down at start) — rate-limit layer degrades to bounded fail-open; idempotency layer fails closed (503); a lock whose TTL expires mid-send cannot be stolen into a double-send (upstream deadline < lock_ttl).
• Contract: DisallowUnknownFields rejects every disallowed field (title, body, image_url, media_id, username, server_url, …) with 400 unexpected_field.
• Contract: allowlist rejections produce exact codes topic_not_allowed / project_not_allowed / package_not_allowed.

Acceptance criteria.

• A request with a valid key, allowlisted topic, and Idempotency-Key reaches the (stub) handler and returns accepted.
• Duplicate idempotency key replays the identical body; in-flight → 409; payload mismatch → 422; missing → 400.
• Over-limit → 429 with relative Retry-After; Redis-down rate-limit fails open, idempotency fails closed.
• Every unknown field and every non-allowlisted topic/project/package is rejected with the exact §5/§8 error code.

---

Phase 3 — APNs client + POST /v1/apple/send (size: L) · gated on Apple provisioning

Goal. A production APNs client (cached ES256 JWT, reused HTTP/2 pool, correct reason→error mapping) and the live /v1/apple/send endpoint building the §5.1 payloads via pushwire, validated against the APNs sandbox.

Files to create/modify.

• internal/apns/client.go — the APNsClient interface + concrete impl (apns2 or hand-rolled per §3 decision).
• internal/apns/jwt.go — ES256 JWT lifecycle (cache, ~50min refresh, never <20min, eager refresh on ExpiredProviderToken).
• internal/apns/transport.go — per-team HTTP/2 pool with ReadIdleTimeout/PingTimeout; env→host selection.
• internal/apns/errors.go — APNs status+reason → caller error.code + retryability (spec §6.4 table).
• internal/pushwire/apns.go — build private_alert / background_wake aps+silo payloads and apns-* headers (spec §5.1).
• internal/httpapi/apple.go — replace the Phase 2 stub with the real send.

Implementation notes.

• JWT (spec §6.1, reference §1.1): ES256 only; header {"alg":"ES256","kid":<10-char Key ID>}, claims {"iss":<10-char Team ID>,"iat":<epoch s>}. Cache one JWT behind a mutex; regenerate on a ~50-min timer, never <20-min apart; eager regenerate + single in-process retry on 403 ExpiredProviderToken; back off / slow refresh on 429 TooManyProviderTokenUpdates. Per-team pool keyed by team even though v1 has one team.
• Single-flighted JWT regeneration (stampede guard). Under a burst of concurrent sends sharing one cached JWT, an expiry (or a 403 ExpiredProviderToken) is observed by many in-flight goroutines at once; a naive per-goroutine regenerate-on-403 would trip 429 TooManyProviderTokenUpdates. Regeneration must therefore be single-flighted (e.g. golang.org/x/sync/singleflight or a generation/epoch counter under the mutex): a goroutine seeing ExpiredProviderToken first checks whether the JWT was already refreshed since its request started and reuses the new token rather than triggering another refresh. Exactly one refresh happens per ~20-min window regardless of how many concurrent 403s arrive.
• Endpoint (spec §6.2): production → https://api.push.apple.com; sandbox → https://api.sandbox.push.apple.com. Both api.sandbox.push.apple.com and the older alias api.development.push.apple.com are valid, equivalent sandbox hosts that still resolve — we standardize on api.sandbox.push.apple.com (the current canonical name and apns2 v0.25.0's HostDevelopment); 02's use of api.development.* is not a discrepancy to resolve, just the older alias. The external contract "environment":"sandbox" is unchanged. The upstream hosts are hardcoded constants never derived from request input. Port 443, 2197 fallback.
• Connection (spec §6.3, reference §3.3): one client/pool per process per env per team; never per push. Non-zero ReadIdleTimeout(~15s) + PingTimeout(~15s). Do not hardcode SETTINGS_MAX_CONCURRENT_STREAMS. Handle GOAWAY (drop+reconnect). Ensure system trust store includes USERTrust RSA (default in OS/Go pools).
• Headers (spec §5.1 table): apns-topic = allowlisted topic; apns-push-type = alert (private_alert) / background (background_wake); apns-priority = 10 / 5 (background must be 5; 10 is an error — reference §1.4.1); apns-collapse-id = collapse_id if present (≤64 bytes); apns-id omitted (Apple generates, echoed back). apns-expiration ships with a finite default (configurable) rather than 0/omitted: a few hours for private_alert, a short TTL for background_wake. Omitting it makes Apple store-and-retry an undelivered push for up to 30 days (reference §1.4) — for a notification whose content is fetched live on wake, a wake fired days later is useless or confusing. Mirror this on FCM with a finite android.ttl (default is otherwise ~4 weeks) for cross-provider parity. (Resolved open Q#5; the exact default values want product input but v1 ships a sane non-30-day default, not an omission.)
• Payloads (pushwire, spec §5.1): private_alert → aps.alert with title-loc-key=SILO_NOTIFICATION_TITLE, loc-key=SILO_NOTIFICATION_GENERIC_BODY, sound=default, plus silo dict (v:1, wake:"notifications.changed", server_device_id, delivery_id); add aps.badge only if badge is present, non-null, and within a bounded range (0–9999) — out-of-range → 400 invalid_field (it is the one numeric field that escapes the opaque-ID discipline, so it is bounded rather than copied verbatim). background_wake → aps.content-available:1 + silo dict. Uncompressed JSON, far under 4 KB; defensive 400 payload_too_large if it ever exceeds.
• Response → caller (spec §5.1): 200 → { "request_id", "apns_id" (Apple's apns-id header), "status":"accepted" }. Error mapping per spec §6.4 (e.g. BadDeviceToken/InvalidToken/DeviceTokenNotForTopic/MissingDeviceToken → 422 bad_device_token; Unregistered/ExpiredToken → 410 unregistered with timestamp passed through in error.message; 403 InvalidProviderToken/etc → 502 upstream_auth_error + ops alert; 429 TooManyRequests → 429 rate_limited + Retry-After; 5xx → 503 upstream_unavailable). InvalidToken is treated equivalently to BadDeviceToken (token-purge, non-retryable) — Apple has migrated some responses from BadDeviceToken to InvalidToken, and the 02 contract requires treating them equivalently; omitting it would let a dead device fall through to the catch-all and never be disabled. Add a default/fallback rule: any unrecognized 400/403/410 reason maps to a safe non-retryable 502 upstream_error rather than being dropped. Connection-severed sends — a request whose HTTP/2 stream is terminated by GOAWAY or any connection reset before a final APNs response arrives → 503 upstream_unavailable (retryable), the same as a network timeout; the relay does not transparently retry it (the caller's outbox does), preserving at-most-once semantics under the idempotency key. Tag the op-log outcome / metric label connection_severed so these are distinguishable. Environment/token mismatch caveat: a sandbox/production mismatch (e.g. a prod token sent with environment=sandbox) presents upstream as BadDeviceToken → bad_device_token, which instructs the caller to purge a perfectly good token; emit a distinct op-log/metric so a systemic misconfig (many bad_device_token from one account) is distinguishable from genuinely dead tokens. Network/timeout to APNs → 503 upstream_unavailable. The relay does not retry transient upstream errors across caller requests (the caller's outbox does); the only in-process retry is the JWT-expiry one (single-flighted — see JWT note below).

Tests.

• Unit: JWT signs valid ES256 with correct header/claims; refresh-timer logic (≥20min, ≤60min, eager-on-expired); single retry on ExpiredProviderToken.
• Concurrency: force JWT expiry under a burst of concurrent sends; assert exactly one regeneration (single-flight) and zero 429 TooManyProviderTokenUpdates.
• Unit (pushwire): golden-file payloads + headers for private_alert and background_wake, with and without badge/collapse_id (byte-exact vs spec §5.1). Same golden files mirrored into silo-server.
• Unit: every row of the spec §6.4 reason→error table maps correctly (table-driven), including InvalidToken → bad_device_token (equivalent to BadDeviceToken) and the default/fallback rule (unrecognized reason → 502 upstream_error, non-retryable).
• Fake APNs upstream injecting a GOAWAY / mid-stream reset before a final response → caller gets 503 upstream_unavailable, the op-log/metric is tagged connection_severed, and there is no lost or double send (the relay does not transparently retry).
• Integration (Apple sandbox, behind -tags=integration, requires .p8): real send to api.sandbox.push.apple.com returns 200 + apns_id; an intentionally bad token returns the expected bad_device_token/unregistered mapping.
• Integration: relayctl ping-upstream --provider apns --env sandbox signs+connects without sending content; optionally captures the sandbox-only apns-unique-id response header (reference §1.6) into the ping output for Push-Notifications-Console delivery-log correlation during client QA.

Acceptance criteria.

• /v1/apple/send builds the exact §5.1 payloads/headers and returns {request_id, apns_id, status:"accepted"} on success against sandbox.
• Background pushes always send apns-priority: 5 and apns-push-type: background.
• JWT is cached, reused, and refreshed on the ~50-min cadence; ExpiredProviderToken triggers one regenerate+retry; concurrent 403s yield exactly one single-flighted regeneration.
• Every spec §6.4 mapping is covered by a passing table-driven test, including InvalidToken and the unrecognized-reason fallback; a GOAWAY/connection-severed send maps to 503 upstream_unavailable with no double-send; the pushwire golden files match across both repos.

---

Phase 4 — FCM client + POST /v1/fcm/send (size: L) · gated on Firebase provisioning

Goal. A production FCM HTTP v1 client (cached OAuth2 token, tuned HTTP/2, correct status→error mapping) and the live /v1/fcm/send endpoint building the §5.2 data-only payloads via pushwire, validated with FCM validateOnly:true.

Files to create/modify.

• internal/fcm/client.go — the FCMClient interface + concrete impl (x/oauth2/google path per §3).
• internal/fcm/oauth.go — CredentialsFromJSONWithType(ctx, json, google.ServiceAccount, "https://www.googleapis.com/auth/firebase.messaging") → cached/auto-refreshing TokenSource, one per project.
• internal/fcm/transport.go — tuned HTTP/2 client; endpoint POST https://fcm.googleapis.com/v1/projects/{PROJECT_ID}/messages:send.
• internal/fcm/errors.go — FCM error.status → caller error.code + retryability + Retry-After honoring (spec §7.4).
• internal/pushwire/fcm.go — build data-only message + android config (spec §5.2).
• internal/httpapi/fcm.go — replace the Phase 2 stub with the real send.

Implementation notes.

• OAuth2 (spec §7.1, reference §2.2): least-privilege scope firebase.messaging; TokenSource caches + auto-refreshes (~1h tokens; never mint per request). On GCP prefer keyless Workload Identity / attached SA; off-GCP load SA JSON from the secret manager (open Q#4). One token source per Firebase project.
• Endpoint (spec §7.2): global host only — no regional endpoints (reference §2.3). The host fcm.googleapis.com is a hardcoded constant never derived from request input. {PROJECT_ID} is the allowlist-checked project_id, and project_id/package_name are validated against a strict charset (e.g. ^[a-z0-9-]+$ / reverse-DNS) before allowlist lookup and URL interpolation, so an allowlist row can never introduce a path-traversal or host-override (no SSRF / path injection).
• Payload — data-only (pushwire, spec §5.2): message.token = caller token; message.data is a fixed, closed set of string-valued keys only ("v":"1", "wake":"notifications.changed", "server_device_id", "delivery_id") — no caller-controlled key or value beyond the opaque IDs; no notification block; message.android.priority = "HIGH" (private_data) / "NORMAL" (background_wake); message.android.collapseKey set only if the caller sent collapse_key. Field-casing reconciliation (reference §2.4): the relay POSTs raw JSON to the real FCM HTTP v1 REST endpoint, whose AndroidConfig field is camelCase collapseKey (FCM v1 rejects unknown fields with INVALID_ARGUMENT, so a literal collapse_key inside android would fail every collapsible send). The inbound caller request body field stays collapse_key per 03; only the relay-built upstream JSON uses collapseKey. Likewise android.priority is sent as canonical uppercase "HIGH"/"NORMAL" (reference §2.4 lists these as the enum values, with lowercase also accepted; the relay sends the canonical uppercase to be robust against future enum tightening). Honor the 03 contract's priority: high-equivalent ("HIGH") for private_data (do not second-guess — spec §5.2 reconciliation note). Set a finite android.ttl (default ~4 weeks otherwise) for parity with the APNs apns-expiration default (above). Never set topic/condition. ≤4096 bytes. Use validateOnly:true (camelCase REST field — reference §2.5) only for the relayctl test path, never production.
  • badge has no data-only FCM home. The relay-built FCM payload is strictly data-only (no notification block), so a caller-provided badge has nowhere legitimate to go. Drop badge from the FCM accepted schema (it is disabled-by-default and would otherwise have to be smuggled into the closed data map, weakening the allowlist). On the APNs side, bound badge to a sane range (e.g. 0–9999) and reject otherwise with 400 invalid_field — it is the one numeric field that escapes the opaque-ID discipline.
  • Collapse-key coalescing is best-effort (reference §2.7). FCM stores at most 4 distinct collapseKeys per device while offline; a 5th evicts one nondeterministically. Since collapse_key is a per-series HMAC and a device may follow >4 series, collapsible coalescing is best-effort and excess distinct keys evict nondeterministically — acceptable because the durable inbox + sync (the caller's outbox) covers any dropped collapsible wake. APNs has no equivalent documented per-device collapse-id cap (note the asymmetry).
• Response → caller (spec §5.2): 200 → { "request_id", "fcm_message_name" (FCM v1 name), "status":"accepted" }. Error mapping per spec §7.4 (e.g. INVALID_ARGUMENT → 422 invalid_argument; SENDER_ID_MISMATCH → 403 sender_id_mismatch; UNREGISTERED → 410 unregistered; QUOTA_EXCEEDED → 429 rate_limited honoring upstream Retry-After else default 60s; INTERNAL/UNAVAILABLE → 503 upstream_unavailable honoring Retry-After on 503; UNSPECIFIED_ERROR → 502 upstream_error, no auto-purge). Classify on error.status/FcmError.errorCode, not the bare HTTP code. Retry-After forwarded to the caller as relative delay-seconds (reference §4.2). Add a default/fallback rule so any unrecognized FCM error.status/errorCode maps to a safe non-retryable 502 upstream_error (never an auto-purge), rather than being dropped.
  • THIRD_PARTY_AUTH_ERROR (401) — re-classified (do NOT page as a relay-credential failure). Per Firebase's error-codes page this means an invalid/missing APNs cert/key or Web Push auth credential and is returned only for iOS-via-APNs or Web Push registration tokens. /v1/fcm/send sends only Android registration tokens (message.token, package com.continuum.app.android, data-only), so a legitimate Android send can never produce THIRD_PARTY_AUTH_ERROR. If it ever appears, treat it as a per-token/registration anomaly → 502 upstream_error, surface for investigation, and do not page on-call as a relay .p8/SA misconfig. The relay's actual OAuth2/service-account credential failures surface as a non-200 from oauth2.googleapis.com/token at mint time, or HTTP 401 UNAUTHENTICATED / 403 PERMISSION_DENIED (the status, not an FcmError code) on messages:send — those map to 502 upstream_auth_error + ops alert.
  • Per-device rate exceedance (DeviceMessageRateExceeded, reference §2.7). The Android per-device caps (240/min, 5,000/hr) are distinct from project-level QUOTA_EXCEEDED and surface defensively as a quota/INVALID_ARGUMENT-adjacent signal whose exact code/string is not firmly documented; detect on error.status/errorCode and map to a retryable 429 rate_limited with a conservative default Retry-After, not a token purge.
  • UNREGISTERED caller-facing 410 is an intentional cross-provider normalization. FCM natively returns UNREGISTERED as HTTP 404; the relay remaps it to caller 410 unregistered so the caller's device-disable ("purge dead token") logic is uniform with the APNs path (which natively uses 410). This is a deliberate divergence from FCM's native status, not a bug.

Tests.

• Unit: token source caches; no per-request mint; refresh margin.
• Unit (pushwire): golden-file data-only payloads for private_data and background_wake, with/without collapse_key (byte-exact vs spec §5.2; all data values strings; no notification block). Mirrored into silo-server.
• Unit: every row of the spec §7.4 status→error table (table-driven), including Retry-After honoring + 60s default, the unrecognized-error.status fallback (502 upstream_error), the per-device-rate (DeviceMessageRateExceeded) → 429 rate_limited (no purge), and THIRD_PARTY_AUTH_ERROR → 502 upstream_error (per-token anomaly, not paged as relay-credential).
• Unit/contract: the FCM upstream JSON uses camelCase collapseKey, uppercase priority ("HIGH"/"NORMAL"), and validateOnly; badge is rejected from the FCM schema; the data map is the closed key set (no extra keys).
• Integration (FCM validateOnly:true, -tags=integration, requires SA JSON): a valid request validates OK; a malformed token surfaces invalid_argument/unregistered mapping.
• Integration: relayctl ping-upstream --provider fcm --env sandbox mints a token + does a validateOnly send.

Acceptance criteria.

• /v1/fcm/send builds the exact §5.2 data-only payload (string values, no notification) and returns {request_id, fcm_message_name, status:"accepted"}.
• OAuth2 token is cached/refreshed, never minted per request; one source per project.
• Every spec §7.4 mapping is covered (incl. the unrecognized-status fallback, per-device-rate, and THIRD_PARTY_AUTH_ERROR as a non-paging per-token anomaly); QUOTA_EXCEEDED/UNAVAILABLE forward Retry-After (default 60s on 429).
• The upstream FCM JSON uses collapseKey (camelCase), "HIGH"/"NORMAL" priority, validateOnly (camelCase), a finite android.ttl, and the closed data key set; badge is not accepted on the FCM path.
• validateOnly test path works; production sends never set validateOnly.

---

Phase 5 — Observability, redaction, metrics (size: M)

Goal. Production-grade redacted logging, op-log persistence, the full Prometheus metric catalog, and /readyz//metrics wired and network-scoped.

Files to create/modify.

• internal/observability/redact.go — redaction middleware (strips Authorization/bearer, raw key, raw token, JWT, OAuth2 token, payload bodies).
• internal/observability/metrics.go — Prometheus registry + the spec §13.2 catalog.
• internal/httpapi/health.go — /readyz (PG SELECT 1, Redis PING; APNs and FCM reported per-provider, not as a single combined gate) and /metrics (promhttp.HandlerFor), both bound to the internal network.
• internal/oplog/gc.go + cmd/relayctl logs --gc subcommand (or a background sweeper in cmd/relay) — the retention GC deliverable; plus the §8.1 partitioning DDL (or a documented partition/offload decision).
• Wire oplog.Write into every send path with the redacted subset.

Implementation notes.

• Redaction enforced in middleware so no code path can leak (spec §11/§12/§13.1). Never log: bearer header, raw API key (log prefix or hash only), raw device tokens (log SHA-256 truncated hash only), payload bodies, relay JWT, OAuth2 tokens. Do log: request_id, account_id, status_code, error_code, upstream reason, token_hash, egress_ip, latency_ms.
• Op-log retention is a concrete Phase 5 deliverable (spec §8.1/§13.1): every send writes one relay_op_logs row (redacted by construction). Ship a real GC — add relayctl logs --gc (to the §5.6 command table and Phase 1/5 file lists) or a built-in background sweeper in cmd/relay — with a definite retention query (DELETE success rows >14d, failure/reject rows >90d) on a schedule with an owner. Either add time-partitioning DDL to the §8.1 schema or document an explicit partition/offload decision here (do not defer it to an open question). Add a disk-usage / relay_op_logs table-growth alert (at 50k/day/account the table grows unbounded without GC — a classic "works for months then the disk fills" surprise).
• Metrics catalog (spec §13.2): relay_requests_total, relay_request_duration_seconds, relay_upstream_requests_total, relay_upstream_duration_seconds, relay_auth_failures_total, relay_rate_limited_total (incl. kind="daily"), relay_rate_limit_degraded_total, relay_idempotency_total, relay_apns_jwt_refresh_total, relay_fcm_token_refresh_total, relay_redis_up, relay_pg_up, plus relay_provider_healthy{provider=...}, relay_credential_healthy{provider=...}, and relay_fcm_project_quota_headroom. Keep labels low-cardinality (no token, no request_id).
• /readyz — per-provider, not all-or-nothing. Return 200 only when the core deps (PG + Redis) are reachable, and report per-provider APNs/FCM health separately in checks (a relay_provider_healthy{provider} gauge). A single provider's credential/init failure must not down the whole replica (that would pull a healthy APNs replica from the LB just because FCM creds are temporarily unresolvable, killing Apple push too); instead the send handler for an un-initialized/degraded provider returns 503 upstream_unavailable for that provider only. Surface per-provider circuit-breaker state so operators see "APNs degraded, FCM healthy" rather than a binary not-ready. Body shape: 503 {"status":"not_ready","checks":{...}} only when a core dep is down (spec §5.4).
• /readyz and /metrics bound to the internal/scrape network; only /v1/* and /healthz are public (spec §11/§14.1).

Tests.

• Unit: redaction middleware drops every disallowed field even when handlers attempt to log them (assert no token/bearer/JWT/OAuth2 string appears in captured log output).
• Unit: a success and a failure path each produce exactly one op-log row with the redacted subset and a token_hash (never raw token).
• Unit: metric counters increment with correct labels for first/replay/conflict idempotency, auth-failure reasons, rate-limit kinds, JWT/token refresh results.
• Integration: /readyz flips to 503 when PG or Redis is down; /metrics exposes the full catalog.

Acceptance criteria.

• No log line or op-log row can contain a raw token, bearer header, JWT, OAuth2 token, or payload body (asserted by tests).
• All spec §13.2 metrics are exported with low-cardinality labels.
• /readyz reflects real dependency + credential health; /metrics and /readyz are not publicly routable.

---

Phase 6 — Hardening, load test, deploy (size: L) · gated on provisioning + secret manager

Goal. Production hardening (real secret-manager wiring, TLS, anti-abuse), a hundreds-of-users burst load/pacing test, idempotency-under-concurrency proof, container build, and rollout.

Files to create/modify.

• internal/config/secrets_*.go — real secret-manager integration (KMS/Vault/cloud secret manager); load .p8, SA JSON, pepper, DSNs at startup into memory; optional SIGHUP re-read.
• deploy/Dockerfile (distroless/minimal), deploy manifests, TLS/HSTS config.
• test/load/ — k6/vegeta burst scenario + concurrency idempotency test harness.
• Alerting rules (Prometheus → alertmanager / SLO doc).

Implementation notes.

• Secrets (spec §11, reference §4.5): credentials loaded once at startup from the secret manager into memory; never on disk in the image/repo, never logged, never returned by any endpoint. On GCP prefer keyless FCM (Workload Identity). Rolling restart picks up rotated secrets.
• TLS (spec §11/§14.1): HTTPS/HTTP-2 only, TLS 1.2+ (prefer 1.3), HSTS on relay.silo.app. Terminate at LB or in-process.
• Anti-abuse (spec §8.2/§11): per-IP auth-failure rate limiting keyed on the trusted-proxy-resolved client IP (caller-supplied X-Forwarded-For/X-Real-IP ignored — see Phase 2 client-IP resolution; the trusted-proxy CIDR list is config); soft lockout (latency / Retry-After) to avoid CGNAT/VPN collateral; body size cap; explicit timeouts; graceful shutdown. Document a response runbook for the key-guessing alert.
• Credential-health validation & rotation runbook (spec §4.4/§14.3). Add a startup + periodic synthetic credential check that proves the loaded credential actually authenticates upstream — not just that a JWT signs / a token mints locally (which /readyz already checks). For FCM, a validateOnly:true send; for APNs, an auth-exercising send to a known-bad token that still distinguishes an auth-layer 403 (InvalidProviderToken) from a token-layer 400 (BadDeviceToken). Wire the result into a relay_credential_healthy{provider=...} gauge and into /readyz's per-provider health (below). Schedule relayctl ping-upstream as a cron for continuous health, and alert on the first upstream_auth_error (§6.4) — Apple actively revokes leaked .p8 keys and SA keys/bindings can be rotated out from under the relay. Document the rotation runbook (overlap window, dual-key support — APNs allows multiple keys and the relay pools by team).
• Graceful-shutdown drain ordering (spec §14.2). On SIGTERM: (1) flip /readyz to 503 so the LB stops routing new requests; (2) keep serving in-flight handlers; (3) bound srv.Shutdown(ctx) to less than lock_ttl so any marker left by a killed-mid-flight handler expires quickly — and release/downgrade the in-flight idempotency marker on handler context-cancellation (Phase 2), so a drained-but-incomplete send frees its lock immediately rather than forcing the caller's retry to eat a 30s 409 idempotency_conflict. A clean drain must leave no orphaned in-flight idempotency markers; the relationship between the shutdown grace period and lock_ttl is explicit.
• Load/pacing test (spec §13 / testing strategy §5 below): simulate a hundreds-of-users burst (e.g. 300–500 concurrent device sends), assert the relay stays under the per-account cap, returns correct 429 + relative Retry-After, and one well-tuned upstream connection sustains the throughput (reference §1.3 — thousands/sec). Verify the FCM smooth-traffic guidance does not trip quota (reference §2.7).
• Idempotency under concurrency: fire N concurrent requests with the same Idempotency-Key; assert at most one upstream send (count via a fake upstream), the rest replay or 409.
• FCM quota-storm shedding: when the shared project quota is exhausted, the relay-side project circuit breaker sheds with 503 upstream_throttled before forwarding (Phase 2 / §6.4), rather than hammering FCM into a 429 stampede.
• HA (spec §14.2): multiple stateless replicas behind the LB; rolling deploys; kill-safety (a drained/killed replica is covered by the caller's outbox retry). Relay-down delays, never loses.

Tests.

• Load: hundreds-of-users burst (latency p50/p95/p99, error rate, 429 correctness).
• Concurrency: same-key idempotency → at-most-once upstream (assert against a counting fake).
• Security: secret-manager load path; no secret in image (docker history/scan); TLS config (min version, HSTS).
• Security: forged caller X-Forwarded-For/X-Real-IP does not change the resolved client IP / egress_ip (only the trusted-proxy header from an allowlisted peer is honored); per-IP throttle cannot be bypassed.
• Drain: rolling-restart drains in-flight requests via srv.Shutdown without dropping a completed send, /readyz flips to 503 before draining, and no orphaned in-flight idempotency markers remain afterward (an immediate caller retry does not get a spurious 409).
• Resilience: FCM quota-exhaustion → relay sheds with 503 upstream_throttled (project circuit breaker) instead of forwarding into a 429 storm.
• Credential health: the synthetic startup/periodic check flips relay_credential_healthy and /readyz when the loaded credential is rejected upstream (auth-layer 403), distinct from a token-layer 400.

Acceptance criteria.

• Secrets load exclusively from the secret manager; image/repo scans show no embedded .p8/SA JSON/pepper.
• A hundreds-of-users burst is handled within the per-account caps with correct 429/Retry-After; one upstream connection sustains the load.
• Concurrent identical-key sends produce at most one upstream call.
• A rolling restart drains in-flight requests, flips /readyz to 503 first, and leaves no orphaned in-flight idempotency markers.
• The synthetic credential check gates /readyz/relay_credential_healthy on real upstream acceptance; the first upstream_auth_error pages ops; the rotation runbook is documented.
• Forged caller forwarding headers cannot poison egress_ip or bypass the per-IP throttle.
• TLS 1.2+ enforced, HSTS set; /metrics+/readyz internal-only; staging deployment validated before production rollout.

---

5. Testing Strategy

5.1 Layers

Layer	What it covers	Provider creds?
Unit	Field validation, JWT/token lifecycle math, key hashing/compare, error-table mappings, redaction, metric labels	No
Contract	Exact 02/03 request/response shapes: DisallowUnknownFields rejects every disallowed field; success/error bodies are byte-exact; pushwire golden files	No
Integration	Real upstream behavior: APNs sandbox sends; FCM validateOnly:true validation; PG migrations; Redis rate-limit/idempotency	APNs .p8 / FCM SA (sandbox)
Redaction assertions	No raw token/bearer/JWT/OAuth2/payload ever reaches a log or op-log	No
Load / pacing	Hundreds-of-users burst; per-account cap + 429/Retry-After; upstream connection reuse sustains throughput	Optional (fake upstream)
Idempotency-under-concurrency	Same key, many concurrent requests → at-most-once upstream send	No (fake upstream)
Failure-mode	APNs GOAWAY/mid-stream reset mapping (no lost/double send); JWT-expiry single-flight (one regen, zero TooManyProviderTokenUpdates); Redis dropped mid-request (idempotency fail-closed, rate-limit bounded fail-open); graceful-drain leaves no orphaned idempotency lock; FCM quota-exhaustion shedding	No (fakes)

5.2 Contract tests against the 02/03 shapes

For each endpoint, a golden corpus of requests/responses derived from spec §5.1/§5.2:

• Valid private_alert / background_wake (APNs) and private_data / background_wake (FCM) requests → assert the built upstream payload + headers are byte-exact vs the spec (the pushwire golden files).
• Every disallowed field (title, body, image_url, media_id, username, server_url, notification, …) → 400 unexpected_field.
• Every validation failure → its exact error.code (invalid_token, invalid_environment, invalid_mode, invalid_collapse_id, topic_not_allowed, project_not_allowed, package_not_allowed, missing_idempotency_key, …).
• The same pushwire golden corpus is shared with silo-server's custom_apns/custom_fcm tests. Because a copied package in one repo cannot fail the other repo's CI on its own, the mirror is a checked-in generated artifact with a committed checksum that both repos verify in CI against the canonical hash (see §3 sync runbook) — a divergence fails CI on both sides regardless of which repo changed first.
• Closed-allowlist assertion (privacy guarantee): a dedicated test asserts the FCM data map keys and the APNs payload key set are exactly the closed allowlist, so adding any key (e.g. smuggling content into data) fails CI even if golden files are regenerated. The content-free guarantee is enforced structurally, not only by golden equality.

5.3 Integration against sandbox / validateOnly

• APNs sandbox (api.sandbox.push.apple.com): a real send returns 200 + apns_id; a deliberately invalid token returns the mapped bad_device_token / unregistered. Run behind -tags=integration, only when .p8 secrets are present (CI integration job).
• FCM validateOnly:true (reference §2.5): a valid request validates without delivering; a malformed token surfaces invalid_argument/unregistered. Never validateOnly on production sends.

5.4 Redaction assertions

Capture log + op-log output across success and every error path; assert via regex that none of: the bearer value, the raw rk_ secret, a raw device token, the relay JWT, or an OAuth2 access token ever appears, and that token_hash is present where a token was involved (spec §12/§13.1).

5.5 Load / pacing test (hundreds-of-users burst)

Simulate a burst of several hundred concurrent device sends (one account, mixed APNs/FCM) against a fake upstream that records call counts and timing:

• Assert per-account rate limiting holds (10 req/s burst + 50k/day; spec §9.2) and over-limit responses are 429 with a relative Retry-After.
• Assert one upstream HTTP/2 connection is reused (no connection-per-push) and sustains the burst (reference §1.3/§3.1).
• Assert latency percentiles and zero double-sends.

5.6 Idempotency-under-concurrency test

Fire N (e.g. 50) concurrent requests sharing one Idempotency-Key at a fake upstream that increments a counter per call. Assert: exactly one upstream call; the other N−1 either replay the stored result or receive 409 idempotency_conflict; a follow-up with a different payload but the same key returns 422 idempotency_key_reuse. This proves at-most-once upstream send under the caller's concurrent outbox retries (spec §10.2).

5.7 Failure-mode tests (the exact modes on-call hits)

Beyond the happy path and the error-table mappings, the plan tests every failure mode that actually pages an engineer (all use fakes — no provider creds):

• APNs GOAWAY / mid-stream reset. A fake APNs upstream injects GOAWAY (or resets the stream) before a final response mid-burst → the caller-facing result is 503 upstream_unavailable, the op-log/metric is tagged connection_severed, and a counting fake confirms no lost or double send (no transparent retry).
• JWT expiry under concurrency. Force the cached APNs JWT to expire under a concurrent burst → assert exactly one single-flighted regeneration and zero 429 TooManyProviderTokenUpdates.
• Redis dropped mid-request (not just down at start) → idempotency fails closed (503); rate-limit degrades to bounded fail-open; a lock whose TTL expires mid-send cannot be stolen (upstream deadline < lock_ttl).
• Graceful-drain idempotency. A rolling restart during in-flight sends leaves no orphaned in-flight idempotency markers (an immediate caller retry does not eat a spurious 409); /readyz flips to 503 before draining.
• FCM quota-exhaustion shedding. With the shared project quota exhausted, the relay-side project circuit breaker sheds with 503 upstream_throttled instead of forwarding into a 429 storm.

---

6. Deployment

6.1 Container build

• deploy/Dockerfile: multi-stage; static relay (and relayctl) binaries on a distroless/minimal base. No credentials baked in (spec §11). Non-root user; read-only filesystem where possible.
• Image carries only the binary + CA roots (ensure USERTrust RSA present — default in ca-certificates; spec §6.3).

6.2 Secret provisioning

Secret	Source	Notes
APNs .p8 + Key ID + Team ID	secret manager (or mounted file)	per official team; never env/image (spec §14.3)
FCM SA JSON or Workload Identity	GCP keyless (on-GCP) / secret manager JSON (off-GCP)	open Q#4 (spec §15)
API-key HMAC pepper	secret manager	rotation is heavier (open Q#6)
Postgres DSN, Redis URL	secret manager	privileged relayctl DSN held only by operators

Rotation: issue a new APNs key / FCM credential, deploy, retire the old; the relay loads secrets at startup so a rolling restart picks them up (spec §14.3). Optional SIGHUP re-read.

6.3 TLS, health, metrics wiring

• TLS 1.2+ (prefer 1.3), HTTP/2, HSTS on relay.silo.app; terminate at LB or in-process (spec §14.1).
• Public: /v1/apple/send, /v1/fcm/send, /healthz. Internal-only: /readyz, /metrics (spec §11/§14.1).
• Liveness probe → /healthz; readiness probe → /readyz; Prometheus scrapes /metrics on the internal network.

6.4 Monitoring / alerting

Alert on (from the §13.2 catalog):

• First upstream_auth_error → page ops. Alert on the caller-facing upstream_auth_error reason (the relay's genuine .p8/SA/OAuth2 credential failure — a non-200 from oauth2.googleapis.com/token, or 401 UNAUTHENTICATED/403 PERMISSION_DENIED/InvalidProviderToken on send) > 0 → credential misconfig, page ops: never caller-fixable. Do NOT key this alert on THIRD_PARTY_AUTH_ERROR, which on this pure-Android path indicates a per-token/registration anomaly, not a relay-credential problem (see Phase 4 reclassification) — surface it for investigation at most, never page.
• Proactive credential-health alert. Alert if the periodic synthetic credential check (Phase 6: a scheduled relayctl ping-upstream / validateOnly FCM send + APNs auth-exercising probe, plus the relay_credential_healthy gauge wired into /readyz) reports the loaded .p8/SA/Workload-Identity credential is no longer accepted upstream — Apple actively revokes leaked .p8 keys and SA keys/bindings can be rotated out from under the relay, so this must fire before production sends start failing.
• Sustained relay_rate_limit_degraded_total increase → Redis degraded (fail-open active; in-process per-account fallback limiter engaged).
• relay_redis_up == 0 / relay_pg_up == 0; any provider showing relay_provider_healthy{provider=...} == 0 → that upstream is degraded (per-provider, not a whole-replica outage).
• Elevated relay_requests_total{status_code="5xx"} rate; relay_request_duration_seconds p99 regression.
• relay_upstream_requests_total{upstream_reason="QUOTA_EXCEEDED"} and low relay_fcm_project_quota_headroom → shared FCM project quota pressure: the relay-side project limiter is shedding with 503 upstream_throttled before hammering FCM (watch Cloud Console; spec §14.5). Treat as a multi-tenant fairness event — one noisy account must not exhaust the ~600k/min project quota shared across all accounts.
• Accounts approaching/hitting the per-account daily cap (relay_rate_limited_total{kind="daily"}) → a legitimate big-release burst can hit 50k/day; surface for an operator to consider a temporary cap bump.
• Anomalous relay_auth_failures_total rate from an egress IP → possible key-guessing (see §6.4 anti-brute-force; the per-IP limiter consumes the resolved client IP, not a caller-supplied header). Note: the fine-grained reason (unknown-prefix vs hash-mismatch) is internal-only and must not be exposed to tenants.

6.5 Rollout

• Staging first (spec §14.4): separate hostname + separate accounts so client QA never touches production quotas. Validate APNs sandbox + FCM validateOnly end-to-end.
• HA / rolling (spec §14.2): multiple stateless replicas behind the LB; rolling deploy with drain (srv.Shutdown); a killed replica is covered by the caller's outbox retry — relay-down delays, never loses.
• Single region in v1, close to PG/Redis (spec §14.5).

---

7. Provisioning Checklist (the long pole — start day 1)

These gate the phases that touch live providers (Phase 3 APNs, Phase 4 FCM, Phase 6 production). Run them in parallel with Phases 0–2, which need no credentials.

7.1 Apple (gates Phase 3 + Phase 6)

• Apple Developer Program account / team with push entitlement (capture the 10-char Team ID).
• Generate an APNs Auth Key (.p8) in the developer portal; capture the Key ID (10 chars) and download the .p8 once (Apple shows it once). Store in the secret manager (spec §14.3).
• Consider Feb-2025 scoped keys (team-scoped or topic-specific) for least privilege (reference §1.2).
• Register per-platform bundle topics for the official build config and seed the account allowlist (spec §8.4): com.continuum.app.ios, com.continuum.app.tvos, com.continuum.app.macos.
• Standardize on the sandbox host api.sandbox.push.apple.com (reference §1.3; api.development.push.apple.com is an equivalent still-resolving alias — no build-time resolution needed) and confirm the APNs server TLS root (USERTrust RSA, present in standard pools; spec §6.3 / reference §1.7).
• Decide apns2 vs hand-rolled client (open Q#3) before Phase 3.

7.2 Firebase / Google (gates Phase 4 + Phase 6)

• Firebase project(s) for the official build config: continuum-prod-android (and a staging project if desired).
• Register the Android package com.continuum.app.android; seed the FCM allowlist (project_id, package_name) (spec §8.4).
• Provision a service account with the least-privilege firebase.messaging scope (reference §2.2). On GCP, prefer keyless Workload Identity / attached SA; off-GCP, download the service-account JSON and store it in the secret manager — never in repo/image (reference §2.2/§4.5; open Q#4).
• Confirm the global send host fcm.googleapis.com (no regional endpoints; reference §2.3) and the default ~600k msgs/min project quota (reference §2.7 / spec §14.5).

7.3 Cross-cutting

• Secret manager chosen and wired (KMS/Vault/cloud secret manager) for .p8, SA JSON, pepper, DSNs (spec §11 / Phase 6).
• DNS + TLS cert for relay.silo.app (and the staging hostname).

---

8. Risks, Mitigations, and Milestone Sequencing

8.1 Risks and mitigations

#	Risk	Likelihood	Impact	Mitigation
R1	Provider provisioning slips (Apple/Firebase approvals) — the real long pole	Med	High (blocks Phase 3/4/6)	Start §7 on day 1 in parallel; Phases 0–2 + all unit/contract tests need no creds; fakes for upstream
R2	sideshow/apns2 is unmaintained (no release since Oct 2024)	Med	Med	Hide behind internal/apns interface; prototype hand-rolled x/net/http2+golang-jwt/jwt/v5 in Phase 3; decide per open Q#3
R3	JWT/token mishandling (over-frequent refresh → APNs 429 TooManyProviderTokenUpdates; per-request mint → throttling; concurrent-403 stampede)	Med	High	Cache + mutex; ~50-min APNs refresh (never <20min), eager-on-expired; single-flighted regeneration so a concurrent burst of ExpiredProviderToken yields exactly one refresh per ~20-min window; OAuth2 TokenSource auto-refresh; metrics relay_apns_jwt_refresh_total/relay_fcm_token_refresh_total
R4	Double-send under concurrent caller retries (incl. lock-TTL expiry mid-send, and stale in-flight markers after a killed/drained replica)	Med	High	Redis SET NX idempotency, fail-closed; pin the upstream call deadline strictly shorter than lock_ttl (≈10 s upstream vs 30 s lock) and refuse to send if the deadline already passed the lock TTL; release/downgrade the in-flight marker on handler context-cancellation; at-most-once test (§5.6); drain test asserting no orphaned in-flight markers (§5.7); key includes attempt_number
R5	Redis outage / eviction	Low	Med	Rate-limit fails open but bounded (in-process per-account fallback limiter, logged + metric); idempotency fails closed (503, retryable via caller outbox); idempotency keyspace requires a non-evicting maxmemory-policy (noeviction) or a dedicated Redis/db so a rate-limit eviction can never silently drop an in-flight idempotency lock (which would violate at-most-once)
R6	Content/identity leak despite design	Low	Critical	Structural: DisallowUnknownFields, content-free schema, data-only FCM, badge-off-by-default, redaction in middleware, op-logs hold only token_hash
R7	Cross-tenant push (one account targeting another's topic/token)	Low	High	Per-account allowlist enforced after auth; namespaced Redis keys; per-account-scoped allowlist cache
R8	Stolen relay API key (incl. abuse-amplification during a Redis outage, and per-IP-throttle bypass via forged forwarding headers)	Low	Med	Revocable keys (relayctl key revoke), per-account rate caps + allowlist bound spend; bounded fail-open (in-process per-account cap when Redis is down — never unlimited); per-IP auth-failure throttling keyed on a trusted-proxy-resolved client IP (caller-supplied X-Forwarded-For/X-Real-IP ignored)
R9	pushwire drift between relay and silo-server	Med	Med	One canonical source repo; mirror is a checked-in generated artifact with a committed checksum verified in CI on both sides; golden-file + closed-allowlist-key tests fail CI on divergence regardless of regeneration order
R10	APNs TLS root rotation (USERTrust)	Low	Low	Use api.sandbox.push.apple.com (the api.development.* alias is equivalent and still resolves — not a risk); rely on the standard trust store (already includes USERTrust RSA)
R11	FCM high-priority deprioritization over 7-day window (reference §2.8)	Low	Low	Honor 03's priority:high for private_data (app surfaces a notification); caller-side priority:normal fallback is the lever (open Q#10)
R12	Op-log volume at 50k/day/account across many accounts (unbounded table → disk fills)	Med	Low	Shipped retention GC (relayctl logs --gc / background sweeper: 14d success / 90d failure) on a schedule with an owner; time-partitioning DDL (or a documented partition/offload decision); table-growth/disk alert

8.2 Milestone sequencing & rough effort

Milestone	Phases	Effort	Provider creds needed	Parallelizable
M0 — Skeleton green	Phase 0	~M (1 wk)	No	—
M1 — Storage + admin	Phase 1	~L (1 wk)	No	with §7 provisioning
M2 — Request path (no upstream)	Phase 2	~L (1 wk)	No	with §7 provisioning
M3 — APNs live (sandbox)	Phase 3	~L (1 wk)	Apple .p8	Phase 3 ∥ Phase 4
M4 — FCM live (validateOnly)	Phase 4	~L (1 wk)	Firebase SA	Phase 4 ∥ Phase 3
M5 — Observability	Phase 5	~M (3 d)	No	—
M6 — Hardened + deployed	Phase 6	~L (1 wk)	All + secret manager	—

Rough total: ~6–7 engineer-weeks of code, with §7 provisioning running in parallel from day 1 as the gating long pole for M3/M4/M6. Phases 3 and 4 can be split across two engineers once Phase 2 lands. The critical path is: M0 → M1 → M2 → (M3 ∥ M4) → M5 → M6, intersected with provisioning availability.

---

9. Open Questions (carried from spec §15)

These do not block starting; resolve at the noted phase.

1. Per-account rate limits — real numbers (resolve from logs post-M6; spec §9.2). v1 ships ~10 req/s burst + 50k/day.
2. APNs sandbox hostname (resolved — note only). Use api.sandbox.push.apple.com; api.development.push.apple.com is an equivalent, still-resolving sandbox alias. The two are interchangeable, so no build-time resolution is required — this is a standardization note, not an open question.
3. apns2 vs hand-rolled APNs client — decide before Phase 3 (§3, reference §3.1).
4. FCM credentials: GCP-keyless vs off-GCP JSON — resolve in deployment design (Phase 6).
5. apns-expiration / android.ttl defaults (largely resolved). v1 ships a finite default (not omitted/30-day): a few hours for private_alert, short TTL for background_wake, with a matching finite FCM android.ttl; both configurable. Only the exact numeric values want product input.
6. API-key hashing: HMAC pepper vs plain salted SHA-256 — plan chooses HMAC pepper; confirm rotation tradeoff (Phase 1).
7. Idempotency-Key spec status — header name + Stripe-style semantics are de-facto; re-check for an RFC.
8. Op-log volume & retention (resolved into a Phase 5 deliverable). Ship relayctl logs --gc (or a background sweeper): 14d success / 90d failure, on a schedule with an owner, plus time-partitioning DDL (or a documented partition/offload decision) and a table-growth alert — not deferred.
9. Multi-region — out of scope for v1; design later if latency to distant self-hosters matters.
10. FCM priority deprioritization — honor 03's priority:high; revisit only if metrics show systematic deprioritization.