Remote Probe Agents

Remote probe agents (gslb-probe) extend Nexus GSLB health and latency vantage beyond the cluster nodes. Drop a small probe-only binary anywhere — an edge VM, a Kubernetes pod, a branch office, a cloud region where no full node runs — and it contributes health verdicts and RTT samples from that vantage into the same aggregation the full nodes use.

Unlike SaaS external-monitoring products (Constellix Sonar, Catchpoint, ThousandEyes), probes are self-hosted and feed directly into GSLB routing decisions.

Licence: pro tier (remote_probes). See Licensing for the per-tier probe cap.

Vantage classes

Every health-reporting endpoint in the cluster is a vantage, of one of two kinds:

Kind	Node ID	Role
`node`	e.g. `lon-01`	Authoritative full node — serves DNS, owns config, votes in quorum.
`probe`	`probe-<name>`	External observer — health + RTT only. Informational for serving decisions; never serves DNS.

Health aggregation is healthy-from-ANY-vantage, so a probe reporting an endpoint healthy keeps it in rotation; a probe reporting it unhealthy does not by itself remove it (a full node still decides). Probe RTTs feed latency routing (GlobalMinRTT) for free — adding probes quietly widens latency-routing coverage.

gslbctl cluster status and the WebUI cluster page list probes separately from nodes, with a last-seen/staleness indicator.

How it works

When remote probes are licensed, each full node projects its health-check config + target IP set into a TTL'd KV bucket (gslb_<cluster>_probe_assignments). This is an ephemeral projection of the git/rqlite-owned config — not durable config — consistent with the NATS "ephemeral only" contract.
gslb-probe connects to the cluster NATS, reads its assignment, builds a health checker over the assigned targets, and publishes health, heartbeat, and RTT into the same subjects/buckets full nodes use, as node probe-<name>.
The (node_id, …) health schema and the healthy-from-ANY aggregation accept the probe vantage with no changes.

The probe hot-swaps its checker whenever the assignment changes, and derives its KV bucket TTLs from the assignment interval.

Running gslb-probe

gslb-probe \
  -nats nats://10.100.0.1:4222 \
  -cluster default \
  -name edge-fra \
  -region eu-central \
  -nats-creds /etc/gslb-probe/nats.creds

The node ID becomes probe-edge-fra. Every flag has an environment-variable equivalent:

Flag	Env	Purpose
`-nats`	`NEXUS_NATS_SERVERS`	comma-separated NATS URLs (required)
`-cluster`	`NEXUS_CLUSTER_ID`	cluster ID; must match gslbd `cluster.id`
`-name`	`NEXUS_PROBE_NAME`	probe name → node ID `probe-<name>` (required)
`-region`	`NEXUS_PROBE_REGION`	informational vantage label
`-nats-creds`	`NEXUS_NATS_CREDS`	NATS credentials file
`-nats-nkey`	`NEXUS_NATS_NKEY`	NATS NKey seed file
`-nats-user` / `-nats-pass`	`NEXUS_NATS_USER` / `NEXUS_NATS_PASS`	NATS user/password
`-nats-ca` / `-nats-cert` / `-nats-key`	`NEXUS_NATS_CA` / `_CERT` / `_KEY`	NATS TLS
`-webhook-allow`	`NEXUS_WEBHOOK_ALLOW`	CIDRs allowed as webhook-check targets (local SSRF allowlist)
`-allow-script-checks`	`NEXUS_ALLOW_SCRIPT_CHECKS=true`	permit `type:script` checks (runs arbitrary commands; default off)

A Kubernetes DaemonSet example (one probe per node) ships at deploy/kubernetes/gslb-probe-daemonset.yaml.

ICMP checks

TCP, HTTP, and webhook checks work with no special privileges. ICMP (ping) checks need CAP_NET_RAW; grant it explicitly (the example DaemonSet drops all capabilities by default).

Security

A probe stamps its own webhook SSRF allowlist (-webhook-allow) and refuses script checks unless -allow-script-checks is set — a compromised assignment writer can neither widen a probe's egress nor make it run arbitrary commands.

Scope probe NATS credentials to health only

Probes must not be able to write anything but health/heartbeat/RTT. Issue probe credentials whose permissions cover only the health subjects and KV buckets, and read-only on the assignment bucket. A representative permission set (tune the exact $JS.API.* subjects to your NATS/JetStream deployment):

publish:
  allow:
    - "gslb.<cluster>.health.probe-*.>"      # own health events
    - "gslb.<cluster>.membership.heartbeat"  # liveness
    - "$KV.gslb_<cluster>_health.>"          # health KV
    - "$KV.gslb_<cluster>_membership.>"      # heartbeat KV
    - "$KV.gslb_<cluster>_rtt.>"             # RTT KV
subscribe:
  allow:
    - "$KV.gslb_<cluster>_probe_assignments.>"  # read assignment (watch)
    - "_INBOX.>"

Grant no access to any other subject or bucket. In this architecture durable config lives in git/rqlite (never in NATS), so a health-scoped probe credential cannot reach configuration at all — but keep the allowlist tight regardless. See the Security Guide for the broader hardening checklist.