Every API replica ran its own enforcer over only its own SessionManager plus
the edge Redis records, because `nodesessions.NewTracker` is constructed only
in proxy/transcode mode — integrated streams never reached the shared
`silo:sessions:` namespace. Replicas were mutually blind, so under-enforcement
of the concurrent-stream cap was certain, and two replicas trimming from
different snapshots could over-kill.
Per decision A6 (Option A):
- `nodesessions.Publisher` mirrors the integrated process's live sessions into
the shared keyspace every 10s. It re-SETs every live record on every tick so
the 60s record TTL is renewed, and diffs only deletions so a stopped session
leaves the shared picture at once rather than lingering for the TTL.
- The publisher's key namespace is derived from a process-unique instance id,
not `resolveNodeIdentity()`. That helper returns SILO_NODE_NAME/NODE_NAME/
hostname, so an operator setting it in shared env and scaling the deployment
would give every replica the same namespace and they would delete each
other's session records — worse than the blindness this fixes.
- `streamenforcer.Coordinator` elects a single evaluator per tick via a
renewable Redis lease (a plain SET NX would lock the holder out of its own
next tick). Each pass is bounded by the evaluation interval so it can never
outlive the lease. No coordinator, or a Redis error, evaluates anyway:
failing to coordinate must never mean failing to enforce.
Also closes the cross-replica gap Batch 3 recorded as an accepted limitation:
`mirrorToRedis` was an unconditional SET, so two replicas revoking the same key
could lose the stronger kill — and edges learn kills only from Redis. It now
merges server-side in Lua with the same two independent monotonic dimensions as
`applyLocal` (expiry never earlier, cutoff never backward, reason follows the
newer cutoff), sharing one Go definition of that comparison.
Two details that are easy to get wrong:
- The merge compares exact (unix_sec, nsec) pairs carried as additive fields on
a dedicated mirror payload. RFC3339 strings cannot be compared
lexicographically (Go omits trailing zeros, so "…:00Z" sorts after
"…:00.5Z"), and millisecond truncation could retain an older cutoff and let a
credential issued between two same-millisecond revocations keep streaming.
- The script merges before deciding to delete. Deleting on a lapsed *incoming*
revocation, as the old unconditional path did, could remove a live permanent
or longer-lived kill written by another replica.
The mirror payload is additive, so old and new nodes interoperate during a
rolling deploy, and any script failure falls back to the previous plain SET
with a once-only warning.
This fixes cross-replica *monitoring* only. Synchronous admission remains
per-process, so streams spread across replicas are still all admitted and are
trimmed asynchronously by the enforcer rather than refused at the door. A
distributed admission gate is separate work.
Part 2 of 3 for the Batch 4 liveness/replica work.
Part of #305
CoffeeKnyte
12f5d5f8ed