State Synchronization (NATS + JetStream)

Nexus GSLB exchanges runtime state (endpoint health and cluster membership) over NATS with JetStream for durability and rejoin resilience. Each node publishes its local active health and heartbeats and subscribes to global state to build a GlobalHealthView. The load balancer can optionally consider the global view via policy.

Concepts

• NATS subjects
  • Health events: gslb.<cluster>.health.<nodeId>.<family> where <family> ∈ {v4,v6}.
  • Heartbeats: gslb.<cluster>.membership.heartbeat.
• JetStream Key-Value (KV)
  • Health bucket: gslb_<cluster>_health (per-IP JSON with TTL).
  • Membership bucket: gslb_<cluster>_membership (per-node JSON with TTL).
• GlobalHealthView
  • Maintains per-IP, per-node health reports with timestamps.
  • Tracks active members by heartbeat TTL.
  • Provides IsHealthy (any active node healthy) and QuorumHealthy helpers.

Policies

• local-or-global-fallback (default): use local health, falling back to the global view.
• prefer-local: return local health; global is used only as a future hint.
• local-only: ignore global view entirely.
• global-any-healthy: healthy if either local or any active global report is healthy.
• global-quorum: healthy if a minimum percentage of active members report healthy within staleness TTL.

Configuration

state:
  healthPolicy: "prefer-local"
  quorumMinPercent: 51
  heartbeatInterval: "10s"
  heartbeatTTL: "30s"
  nats:
    servers: ["nats://n1:4222","nats://n2:4222","nats://n3:4222"]
    tls:
      caFile: "/etc/gslb/pki/ca.crt"
      certFile: "/etc/gslb/pki/client.crt"
      keyFile: "/etc/gslb/pki/client.key"
    auth:
      credsFile: "/etc/gslb/nats/client.creds"   # or user/password or nkey
    jetStream:
      domain: "gslb"

Behavior

• On start, subscriber snapshots both KV buckets (health+membership) then subscribes to subjects for live updates.
• Publisher sends a heartbeat at a fixed cadence and updates KV for the node.
• Health events are pushed immediately on state transitions (healthy→unhealthy or unhealthy→healthy) via a non-blocking channel, keeping peer convergence latency under 200 ms regardless of the health check interval. A 30 s background catchup ticker publishes the full snapshot to recover from any missed signals.
• TTLs: health KV TTL is 2×health.interval by default; membership TTL is configurable.
• During WAN partitions or NATS failures, the node operates with the local health only.

Metrics

• gslbd_state_nats_connected (0/1)
• gslbd_state_nats_published_total{type} and ..._received_total{type} with type ∈ {health, heartbeat}
• gslbd_state_kv_put_total{bucket,result} and ...kv_get_total{bucket,result}
• gslbd_state_merge_lag_ms histogram
• gslbd_state_active_members gauge

Code references

• internal/state/*: NATS client, publisher, subscriber, subjects, types, view, provider.
• cmd/gslbd/main.go: wiring and policy selection.

---

NATS carries ephemeral state only

NATS is used exclusively for ephemeral, TTL'd cluster state. This is a hard architectural boundary. The complete NATS taxonomy is:

Data	Transport	Bucket / subject
Health up/down transition events	JetStream pub/sub	gslb.<cluster>.health.<node>.<v4|v6>
Live health-state snapshot	JetStream KV (TTL)	gslb_<cluster>_health
Node heartbeat / liveness	JetStream KV (TTL)	gslb_<cluster>_membership
Per-node RTT samples (latency routing)	JetStream KV (TTL)	gslb_<cluster>_rtt
Session tokens	JetStream KV (TTL)	(API session store)

What does NOT live in NATS:

• Configuration (YAML) — owned by git, applied per-node via GitOps self-restart. No coordinator.
• Data-plane objects (pools, members, services, users, …) — owned by the rqlite datastore, replicated across nodes via Raft.

Earlier releases distributed config and full DB snapshots over JetStream (the NEXUS.cfg stream / NEXUS_CFG KV bucket and a DB-snapshot bucket). Those paths were removed: rqlite now owns durable cross-node replication, and NATS is strictly ephemeral. If a new feature needs durable cross-node state, it belongs in rqlite, not NATS.