State Synchronization Setup (NATS + JetStream)

This guide walks you through enabling global state synchronization for health and membership across multiple gslbd instances using NATS with JetStream.

Overview

• Each node publishes local health and heartbeats to NATS subjects and KV buckets.
• Health state changes (healthy→unhealthy or unhealthy→healthy) are pushed immediately to NATS peers — peer convergence latency is under 200 ms regardless of checkInterval. A 30 s catchup ticker republishes the full snapshot in the background to recover from any missed signals.
• Each node subscribes to the cluster subjects and KV snapshots to maintain a global view (GlobalHealthView).
• The load balancer uses a composite provider to merge local and global health according to policy.

When to enable

• You run multiple GSLB nodes across regions and want them to share health state.
• You want quorum-based decisions or fast propagation of health changes.

Prerequisites

• NATS 2.x with JetStream enabled.
• TLS/mTLS credentials for clients (recommended).
• A cluster ID configured in gslbd.

Recommended topology (super-cluster)

• 3+ NATS servers per region (odd number for availability).
• Connect regions via NATS gateways to form a super-cluster.
• Enable JetStream in each cluster; optionally set a JetStream domain (e.g., gslb).

Minimal NATS server (single region) example

# nats-server.conf (excerpt)
jetstream: {
  domain: gslb
}
server_name: n1
port: 4222
cluster: {
  name: gslb
  port: 6222
}
# TLS, accounts, and gateways omitted for brevity; see NATS docs

Configure gslbd Add to /etc/gslb/config.yaml:

cluster:
  id: "prod-global"

state:
  healthPolicy: "prefer-local"   # or global-any-healthy | global-quorum
  quorumMinPercent: 51            # used for global-quorum
  heartbeatInterval: "10s"
  heartbeatTTL: "30s"
  nats:
    servers: ["nats://n1.example.com:4222","nats://n2.example.com:4222","nats://n3.example.com:4222"]
    tls:
      caFile: "/etc/gslb/pki/ca.crt"
      certFile: "/etc/gslb/pki/client.crt"
      keyFile: "/etc/gslb/pki/client.key"
    jetStream:
      domain: "gslb"

!!! note "NATS is ephemeral-only" State sync over NATS covers health and membership only — ephemeral, TTL'd data. Configuration and data-plane objects are not synced over NATS: config flows from git via GitOps self-restart, and pools/members/services/users are replicated by the rqlite datastore (Raft). The old JetStream config-sync (enableConfigSync, NEXUS_CFG) has been removed. See State Sync reference.

Policies explained

• local-or-global-fallback (default): use local health, falling back to the global view.
• prefer-local: local result is authoritative; global used mainly for hints.
• local-only: ignore global state (useful during incidents or testing).
• global-any-healthy: consider healthy if any site reports healthy.
• global-quorum: healthy only if a percent of active members report healthy within staleness TTL.

How quorum works

• Each node publishes periodic heartbeats.
• Active membership = nodes whose latest heartbeat is within heartbeatTTL.
• For an endpoint, only reports from active members within heartbeatTTL are counted.
• quorumMinPercent computes healthy threshold: healthy*100/total >= quorumMinPercent.

Verifying connectivity and state

• Metrics to watch:
  • gslbd_state_nats_connected (1 when connected)
  • gslbd_state_active_members
  • gslbd_state_nats_{published,received}_total{type}
  • gslbd_state_kv_{put,get}_total{bucket,result}
  • gslbd_state_merge_lag_ms — histogram of NATS transit latency from publish to peer receipt; p95 > 500 ms indicates network issues between nodes
  • gslbd_health_failover_detection_seconds{direction} — Layer 1 detection latency per state change on this node; direction="down" is healthy→unhealthy
• When Configuration Sync is enabled, logs will include messages like: config snapshot loaded and config event: cluster=... version=....
• Logs will show connect errors or KV bucket creation errors.

Failure modes

• If NATS is unreachable: gslbd continues with local health only (policy prefer-local).
• If clocks skew: events include timestamps; quorum logic tolerates within TTL but keep NTP enabled.
• If membership is zero: no quorum can be established; provider falls back to local (for global-quorum).

Security

• Use mTLS for client/server auth or NATS accounts/JWT with TLS.
• Restrict subjects and KV permissions to the cluster namespace (e.g., gslb.<cluster>.*).
• Rotate client certs periodically; see Security Guide.