Three-Node Cluster (Cloud/VM)

Three-Node Test Cluster

This guide walks through deploying a Nexus GSLB cluster across three geographically distributed nodes using WireGuard for overlay networking and a three-peer NATS JetStream cluster for state synchronisation. It is written for the specific topology of two public VPS nodes and one home-lab node behind NAT, but the pattern applies to any mix of public and private nodes.

Topology

Node	Location	WireGuard IP	DNS public?
`lon-01`	London VPS	10.100.0.1	Yes
`eu-01`	EU VPS	10.100.0.2	Yes
`lab-01`	Home lab VM	10.100.0.3	No (internal)

Each node runs:

gslbd with its own SQLite database
A NATS server (cluster peer)

NATS traffic stays inside the WireGuard overlay. The home lab node connects outbound through WireGuard so NAT traversal is transparent.

Why three full NATS peers and not a leaf node? Three peers give JetStream a 2-of-3 quorum, which means the cluster survives a single node failure while still persisting health events and membership state. A leaf node only relays messages — it does not participate in consensus.

Home lab: VM or Kubernetes? Start with a VM (Docker Compose or bare binary) for the initial cluster test. The Kubernetes manifests in deploy/kubernetes/ can be validated as a second pass once the NATS path is confirmed working.

Phase 1 — WireGuard Mesh

Install WireGuard on all three nodes.

Generate a key pair on each node:

wg genkey | tee /etc/wireguard/privatekey | wg pubkey > /etc/wireguard/publickey
chmod 600 /etc/wireguard/privatekey
cat /etc/wireguard/publickey

Exchange the three public keys, then write the config files.

`lon-01` — `/etc/wireguard/wg0.conf`

[Interface]
Address    = 10.100.0.1/24
ListenPort = 51820
PrivateKey = <LON_PRIVATE_KEY>

[Peer]
# eu-01
PublicKey           = <EU_PUBLIC_KEY>
AllowedIPs          = 10.100.0.2/32
Endpoint            = <EU_VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25

[Peer]
# lab-01 — no Endpoint; lab connects to us
PublicKey           = <LAB_PUBLIC_KEY>
AllowedIPs          = 10.100.0.3/32
PersistentKeepalive = 25

`eu-01` — `/etc/wireguard/wg0.conf`

[Interface]
Address    = 10.100.0.2/24
ListenPort = 51820
PrivateKey = <EU_PRIVATE_KEY>

[Peer]
# lon-01
PublicKey           = <LON_PUBLIC_KEY>
AllowedIPs          = 10.100.0.1/32
Endpoint            = <LON_VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25

[Peer]
# lab-01 — no Endpoint; lab connects to us
PublicKey           = <LAB_PUBLIC_KEY>
AllowedIPs          = 10.100.0.3/32
PersistentKeepalive = 25

`lab-01` — `/etc/wireguard/wg0.conf`

[Interface]
Address    = 10.100.0.3/24
ListenPort = 51820
PrivateKey = <LAB_PRIVATE_KEY>
# No ListenPort — we initiate all connections outbound

[Peer]
# lon-01
PublicKey           = <LON_PUBLIC_KEY>
Endpoint            = <LON_VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25

[Peer]
# eu-01
PublicKey           = <EU_PUBLIC_KEY>
AllowedIPs          = 10.100.0.2/32
Endpoint            = <EU_VPS_PUBLIC_IP>:51820
PersistentKeepalive = 25

Enable and start on each node:

systemctl enable --now wg-quick@wg0

Verify the mesh before continuing — all three nodes must reach each other:

# from lon-01
ping -c 3 10.100.0.2   # eu-01
ping -c 3 10.100.0.3   # lab-01

# check peer status
wg show

Firewall: open WireGuard on VPS nodes

# lon-01 and eu-01
ufw allow 51820/udp

The home lab node opens no inbound ports — it initiates all WireGuard handshakes outbound.

Phase 2 — NATS Cluster

Download the nats-server binary on each node from https://nats.io/download/. Install to /usr/local/bin/nats-server.

Each server listens on its WireGuard IP. The cluster routing port (6222) is never exposed publicly — it is only reachable over the overlay.

`lon-01` — `/etc/nats/nats.conf`

server_name: lon-01
listen:      10.100.0.1:4222
http:        10.100.0.1:8222

cluster {
  name:   nexus-gslb
  listen: 10.100.0.1:6222
  routes: [
    nats-route://10.100.0.2:6222
    nats-route://10.100.0.3:6222
  ]
}

jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

`eu-01` — `/etc/nats/nats.conf`

server_name: eu-01
listen:      10.100.0.2:4222
http:        10.100.0.2:8222

cluster {
  name:   nexus-gslb
  listen: 10.100.0.2:6222
  routes: [
    nats-route://10.100.0.1:6222
    nats-route://10.100.0.3:6222
  ]
}

jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

`lab-01` — `/etc/nats/nats.conf`

server_name: lab-01
listen:      10.100.0.3:4222
http:        10.100.0.3:8222

cluster {
  name:   nexus-gslb
  listen: 10.100.0.3:6222
  routes: [
    nats-route://10.100.0.1:6222
    nats-route://10.100.0.2:6222
  ]
}

jetstream {
  store_dir:       /var/lib/nats/jetstream
  max_memory_store: 256MB
  max_file_store:   1GB
}

systemd unit — all nodes

# /etc/systemd/system/nats.service
[Unit]
Description=NATS Server
After=network.target wg-quick@wg0.service
Wants=wg-quick@wg0.service

[Service]
ExecStart=/usr/local/bin/nats-server -c /etc/nats/nats.conf
Restart=on-failure
User=nats
StateDirectory=nats

[Install]
WantedBy=multi-user.target

useradd --system --no-create-home --shell /usr/sbin/nologin nats
mkdir -p /var/lib/nats/jetstream
chown -R nats:nats /var/lib/nats

systemctl enable --now nats

Verify NATS cluster

Install the nats CLI (https://nats.io/download/), then:

nats --server nats://10.100.0.1:4222 server list
nats --server nats://10.100.0.1:4222 server report jetstream

All three servers (lon-01, eu-01, lab-01) should appear. JetStream should show a single meta leader with two followers.

Phase 3 — gslbd Deployment

Use Docker Compose on all three nodes (including the home lab VM). network_mode: host is required so the container can reach the WireGuard IPs without any special routing.

Directory layout (each node)

/opt/nexus-gslb/
├── docker-compose.yml
├── config.yaml
├── .env
└── data/              ← SQLite volume (created by Docker)

`docker-compose.yml` (same on all nodes)

services:
  gslbd:
    image: registry.starstorm.dev/nexus-gslb/gslbd:latest
    restart: unless-stopped
    network_mode: host
    volumes:
      - ./config.yaml:/etc/gslb/config.yaml:ro
      - ./data:/var/lib/gslbd
    env_file: .env
    healthcheck=:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/api/v1/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 15s

`.env` (each node, do not commit)

GSLB_API_KEY=<strong-random-secret>
GSLB_WS_SECRET=<strong-random-secret>
# License (apply the same floating token to all nodes — see docs/reference/Licensing.md)
GSLB_LICENSE_PUBLIC_KEY=<ed25519-public-key-from-licensectl>
GSLB_LICENSE_KEY=<floating-token-from-licensectl>

Use the same GSLB_API_KEY on all nodes so any node can authenticate API requests from your workstation. Use the same floating license token on all nodes — do not include an Install ID when generating the token for a cluster.

`lon-01` — `config.yaml`

dns:
  listenAddr: "0.0.0.0"
  port: 5353
  domain: "gslb.example.com"

cluster:
  id: "nexus-test"

node:
  id: "lon-01"

rqlite:
  httpAddr: "localhost:4001"

api:
  enabled=: true
  listenAddr: "0.0.0.0"
  port: 8080
  webuiPath: "/opt/gslb/webui"

metrics:
  enablePrometheus: true
  listenAddr: "0.0.0.0"
  port: 9090

health:
  enabled: true
  type: "tcp"
  port: 80
  checkInterval: "10s"
  timeout: "2s"

state:
  nats:
    servers:
      - "nats://10.100.0.1:4222"
      - "nats://10.100.0.2:4222"
      - "nats://10.100.0.3:4222"
  healthPolicy: "prefer-local"
  heartbeatInterval: "10s"
  heartbeatTTL: "30s"

`eu-01` — `config.yaml`

Identical to lon-01 except:

node:
  id: "eu-01"

`lab-01` — `config.yaml`

Identical to lon-01 except:

node:
  id: "lab-01"

dns:
  listenAddr: "127.0.0.1"   # not exposed publicly
  port: "5353"

Start gslbd on each node

cd /opt/nexus-gslb
docker compose pull
docker compose up -d
docker compose logs -f

Phase 4 — Firewall Rules

VPS nodes (lon-01, eu-01)

# WireGuard (already opened in Phase 1)
ufw allow 51820/udp

# DNS — public
ufw allow 5353/udp
ufw allow 5353/tcp

# API — restrict to your own IP or put behind a reverse proxy
ufw allow from <YOUR_IP> to any port 8080

# Metrics — internal only (or Prometheus scraper IP)
ufw allow from 10.100.0.0/24 to any port 9090

# NATS — WireGuard only, no public rule needed
# (already scoped by WireGuard AllowedIPs)

DNS on port 53

The default port is 5353 (no capabilities required). To accept queries on port 53, add a redirect on each VPS:

iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-port 5353
iptables -t nat -A PREROUTING -p tcp --dport 53 -j REDIRECT --to-port 5353
# Persist with iptables-save or nftables equivalent

Phase 5 — Verification

1. Confirm all nodes are healthy

curl http://10.100.0.1:8080/api/v1/health   # lon-01
curl http://10.100.0.2:8080/api/v1/health   # eu-01
curl http://10.100.0.3:8080/api/v1/health   # lab-01

All should return {"status":"ok"}.

2. Check NATS cluster state

nats --server nats://10.100.0.1:4222 server list
nats --server nats://10.100.0.1:4222 server report jetstream

Three peers, one meta leader, JetStream enabled on all.

3. Create a test pool and members on one node

API=http://10.100.0.1:8080
KEY=<your-api-key>

# Create pool
POOL=$(curl -sX POST $API/api/v1/pools \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"test-pool"}' | jq -r .id)

# Add two members — use IPs you can control firewall rules on
curl -sX POST $API/api/v1/pools/$POOL/members \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"host-a","ipAddress":"1.1.1.1","port":80,"weight":1}'

curl -sX POST $API/api/v1/pools/$POOL/members \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"name":"host-b","ipAddress":"1.0.0.1","port":80,"weight":1}'

# Add a health check
curl -sX PUT $API/api/v1/pools/$POOL/healthcheck \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"type":"tcp","port":80,"intervalMs":5000,"timeoutMs":2000}'

4. Test health propagation across nodes

Open a WebSocket listener on two different nodes simultaneously:

# Terminal 1 — watch lon-01
wscat -c "ws://10.100.0.1:8080/api/v1/ws?token=$(
  curl -s http://10.100.0.1:8080/api/v1/ws/token \
    -H "Authorization: Bearer $KEY" | jq -r .token)"

# Terminal 2 — watch eu-01
wscat -c "ws://10.100.0.2:8080/api/v1/ws?token=$(
  curl -s http://10.100.0.2:8080/api/v1/ws/token \
    -H "Authorization: Bearer $KEY" | jq -r .token)"

Block 1.0.0.1 on the firewall of lon-01 (e.g., iptables -A OUTPUT -d 1.0.0.1 -j DROP). Within two health intervals both WebSocket connections should receive a health_update event showing 1.0.0.1 as unhealthy. This confirms:

lon-01's health checker detected the failure
dbSink published to NATS
HealthBridge on both eu-01 and lab-01 received the event and pushed it to their WebSocket clients

Remove the block and verify recovery:

iptables -D OUTPUT -d 1.0.0.1 -j DROP

5. Test `global-quorum` policy

Update one node's config to use global-quorum:

state:
  healthPolicy: "global-quorum"
  quorumMinPercent: 67     # 2 of 3 nodes must agree

With the block on lon-01 only, 1.0.0.1 is unhealthy from lon-01's view but healthy from eu-01 and lab-01. With 67% quorum (2 of 3), the member stays healthy. Block it on two nodes — quorum fails and DNS responses on all nodes stop including that member.

Health Policy Reference

Policy	Behaviour
`prefer-local`	Uses local checker result; global is informational only
`local-only`	Ignores NATS entirely; each node decides independently
`global-any-healthy`	Healthy if local or any active node reports healthy
`global-quorum`	Healthy if ≥ `quorumMinPercent` of active nodes report healthy

prefer-local is the right default for a split-horizon setup where each node checks its nearest backends. Use global-quorum when you want the cluster to agree before withdrawing an endpoint from DNS.

What Is Not Covered Here

Feature	Where to look
Config sync (pool/member replication across nodes)	`docs/StateSync.md` → Configuration Sync section
Geo-IP routing	`docs/Configuration.md` → `algorithm: geo-ip`
DNSSEC	`docs/Security.md`
DNS delegation (pointing a real subdomain at both VPS IPs)	Your registrar + two NS/A records
Kubernetes deployment for the lab node	`docs/user/Kubernetes.md`
Prometheus + Grafana setup	`docs/user/MetricsObservability.md`
TLS on the REST API	Put nginx or Caddy in front; see `docs/Security.md`