CoffeeKnyte
12f5d5f8ed
Every API replica ran its own enforcer over only its own SessionManager plus the edge Redis records, because `nodesessions.NewTracker` is constructed only in proxy/transcode mode — integrated streams never reached the shared `silo:sessions:` namespace. Replicas were mutually blind, so under-enforcement of the concurrent-stream cap was certain, and two replicas trimming from different snapshots could over-kill. Per decision A6 (Option A): - `nodesessions.Publisher` mirrors the integrated process's live sessions into the shared keyspace every 10s. It re-SETs every live record on every tick so the 60s record TTL is renewed, and diffs only deletions so a stopped session leaves the shared picture at once rather than lingering for the TTL. - The publisher's key namespace is derived from a process-unique instance id, not `resolveNodeIdentity()`. That helper returns SILO_NODE_NAME/NODE_NAME/ hostname, so an operator setting it in shared env and scaling the deployment would give every replica the same namespace and they would delete each other's session records — worse than the blindness this fixes. - `streamenforcer.Coordinator` elects a single evaluator per tick via a renewable Redis lease (a plain SET NX would lock the holder out of its own next tick). Each pass is bounded by the evaluation interval so it can never outlive the lease. No coordinator, or a Redis error, evaluates anyway: failing to coordinate must never mean failing to enforce. Also closes the cross-replica gap Batch 3 recorded as an accepted limitation: `mirrorToRedis` was an unconditional SET, so two replicas revoking the same key could lose the stronger kill — and edges learn kills only from Redis. It now merges server-side in Lua with the same two independent monotonic dimensions as `applyLocal` (expiry never earlier, cutoff never backward, reason follows the newer cutoff), sharing one Go definition of that comparison. Two details that are easy to get wrong: - The merge compares exact (unix_sec, nsec) pairs carried as additive fields on a dedicated mirror payload. RFC3339 strings cannot be compared lexicographically (Go omits trailing zeros, so "…:00Z" sorts after "…:00.5Z"), and millisecond truncation could retain an older cutoff and let a credential issued between two same-millisecond revocations keep streaming. - The script merges before deciding to delete. Deleting on a lapsed *incoming* revocation, as the old unconditional path did, could remove a live permanent or longer-lived kill written by another replica. The mirror payload is additive, so old and new nodes interoperate during a rolling deploy, and any script failure falls back to the previous plain SET with a once-only warning. This fixes cross-replica *monitoring* only. Synchronous admission remains per-process, so streams spread across replicas are still all admitted and are trimmed asynchronously by the enforcer rather than refused at the door. A distributed admission gate is separate work. Part 2 of 3 for the Batch 4 liveness/replica work. Part of #305
59 lines
1.3 KiB
Go
59 lines
1.3 KiB
Go
package streamenforcer
|
|
|
|
import (
|
|
"context"
|
|
"time"
|
|
|
|
"github.com/redis/go-redis/v9"
|
|
)
|
|
|
|
const coordinatorKey = "silo:enforcer:leader"
|
|
|
|
var acquireLeaseScript = redis.NewScript(`
|
|
local v = redis.call('GET', KEYS[1])
|
|
if v == false or v == ARGV[1] then
|
|
redis.call('SET', KEYS[1], ARGV[1], 'PX', ARGV[2])
|
|
return 1
|
|
end
|
|
return 0
|
|
`)
|
|
|
|
// Coordinator elects one enforcer replica for an evaluation pass.
|
|
type Coordinator interface {
|
|
Acquire(ctx context.Context) (bool, error)
|
|
}
|
|
|
|
// RedisCoordinator uses a renewable lease to select one evaluator.
|
|
type RedisCoordinator struct {
|
|
rdb *redis.Client
|
|
instanceID string
|
|
leaseTTL time.Duration
|
|
}
|
|
|
|
// NewRedisCoordinator creates a coordinator whose lease lasts twice the
|
|
// evaluation interval.
|
|
func NewRedisCoordinator(rdb *redis.Client, instanceID string, interval time.Duration) *RedisCoordinator {
|
|
if interval <= 0 {
|
|
interval = DefaultInterval
|
|
}
|
|
return &RedisCoordinator{rdb: rdb, instanceID: instanceID, leaseTTL: 2 * interval}
|
|
}
|
|
|
|
// Acquire claims or renews this instance's lease.
|
|
func (c *RedisCoordinator) Acquire(ctx context.Context) (bool, error) {
|
|
if c == nil || c.rdb == nil {
|
|
return true, nil
|
|
}
|
|
result, err := acquireLeaseScript.Run(
|
|
ctx,
|
|
c.rdb,
|
|
[]string{coordinatorKey},
|
|
c.instanceID,
|
|
c.leaseTTL.Milliseconds(),
|
|
).Int()
|
|
if err != nil {
|
|
return false, err
|
|
}
|
|
return result == 1, nil
|
|
}
|