Health Checks

Pleiades performs active health checks against all configured endpoints and exposes the last-known status to the load balancer and (optionally) to the global state sync publisher.

Check Types

Configuration

TCP (default)

health:
  type: tcp
  port: 443
  checkInterval: 10s
  timeout: 2s

HTTP / HTTPS

health:
  type: http            # tcp | http | icmp | script | webhook
  port: 443
  checkInterval: 10s
  timeout: 2s
  http:
    path: "/healthz"
    host: "app.example.com"   # optional: FQDN used for SNI + HTTP Host header
    expectedStatus: 200        # 0 to ignore status code
    contains: "ok"            # empty to ignore body substring check
    tls: true
    insecureSkipVerify: false  # set true only for trusted/self-signed test envs

ICMP Ping

{
  "type": "icmp",
  "icmpCount": 3,
  "intervalMs": 10000,
  "timeoutMs": 2000
}

icmpCount (default 3): number of echo requests sent per probe interval. The check succeeds if at least one reply is received. The RTT recorded is the average of successful replies.

Privilege requirement: ICMP raw sockets require CAP_NET_RAW on Linux. If gslbd runs as root this is automatic. Otherwise grant the capability:

sudo setcap cap_net_raw+ep /usr/local/bin/gslbd

Port is ignored for ICMP checks.

Custom Script

⚠️ Security — script checks execute arbitrary commands as the gslbd user on every node, and scriptContent is replicated cluster-wide via NATS. Treat a script check as remote code execution on your whole fleet. Two controls gate them, both required:

1. Disabled by default. The daemon must opt in with health.allowScriptChecks: true in its config. While disabled, the config validator rejects a script type in the config file, the API refuses to create one (403), and any script check that reaches a node another way (GitOps, snapshot restore, direct datastore write) is not executed — the pool is treated as unconfigured (endpoints healthy by default) and a warning is logged.
2. System administrator only. Even when enabled, the API accepts a script check only from a system administrator (the GSLB_API_KEY holder or a tenant_admin on the default tenant). Tenant/operator API keys are refused (403). This prevents a tenant from executing code on the host.

Prefer tcp/http/icmp/webhook checks wherever possible. Use webhook to delegate custom logic to your own service instead of running code on the node.

{
  "type": "script",
  "scriptPath": "/usr/local/bin/check-myapp.sh",
  "port": 8080,
  "intervalMs": 30000,
  "timeoutMs": 5000
}

The script receives two environment variables:

• Exit code 0 → healthy
• Any other exit code → unhealthy
• Script is killed after timeoutMs milliseconds

Webhook

{
  "type": "webhook",
  "webhookURL": "https://monitor.example.com/check",
  "webhookMethod": "POST",
  "port": 80,
  "intervalMs": 10000,
  "timeoutMs": 3000
}

For POST (default), the daemon sends:

{"ip": "10.0.0.1", "port": 80}

For GET, no body is sent. HTTP 2xx response = healthy; any other status or network error = unhealthy.

The port field is included in the webhook body even for GET requests.

Security — SSRF protection. A webhook check makes gslbd issue a request to a URL you supply, from the node. To stop this being used to reach cloud metadata endpoints or internal services, webhook checks may target public addresses only by default. Loopback, link-local (including the metadata IP 169.254.169.254), private/ULA, and CGNAT destinations are blocked — enforced both when the check is created (400 for a literal private/metadata IP) and at request time on the resolved address (defeating DNS rebinding; redirects are not followed).

To allow a specific internal host (e.g. your own monitoring endpoint on a private network), add its address or CIDR to health.webhookAllowedHosts:

health:
  webhookAllowedHosts:
    - "10.20.0.0/24"      # internal monitoring subnet
    - "192.0.2.50"        # a single host

Entries override the deny list for those ranges only; public destinations remain allowed. webhookURL must be an absolute http/https URL.

Kubernetes and Container Considerations

ICMP (ping) in containers

ICMP requires CAP_NET_RAW. The default K8s statefulset drops all capabilities. You must explicitly re-add NET_RAW:

# In the gslbd container securityContext:
securityContext:
  capabilities:
    drop: ["ALL"]
    add: ["NET_RAW"]

In plain Docker:

docker run --cap-add NET_RAW ... gslbd

If the capability is absent, the ICMP check will log a warning on every probe and mark the endpoint unhealthy — it will never silently pass.

Custom script checks in containers

Script checks are disabled by default and admin-only when enabled — see the security note under Custom Script before using either option below. The daemon needs health.allowScriptChecks: true.

readOnlyRootFilesystem: true only locks the container's overlay filesystem — mounted volumes are still writable. Three approaches work, in order of recommendation:

Option A: scriptContent (recommended — works everywhere)

Store the script body in the health check config via the API, WebUI, or Terraform. gslbd writes it to a tmpfs temp file at probe time, executes it, and removes it. The script is stored in SQLite and replicated to every cluster node via NATS automatically — no filesystem setup required on any node.

{
  "type": "script",
  "scriptContent": "#!/bin/sh\ncurl -sf http://$NEXUS_HC_IP:$NEXUS_HC_PORT/health",
  "intervalMs": 30000,
  "timeoutMs": 5000
}

In Terraform:

resource "nexus_health_check" "app" {
  pool_id        = nexus_pool.app.id
  type           = "script"
  script_content = file("${path.module}/scripts/check-app.sh")
  interval_ms    = 30000
  timeout_ms     = 5000
}

Works identically on VMs, Docker, and Kubernetes. The K8s statefulset includes a tmpfs emptyDir at /tmp for this purpose.

Option B: data PVC (simplest, single-replica)

The StatefulSet already has a PVC mounted at /var/lib/gslbd for the SQLite database. Scripts can live in a subdirectory of that same volume with no manifest changes:

kubectl exec -n nexus-gslb gslbd-0 -- mkdir -p /var/lib/gslbd/hc-scripts
kubectl cp check.sh nexus-gslb/gslbd-0:/var/lib/gslbd/hc-scripts/check.sh
kubectl exec -n nexus-gslb gslbd-0 -- chmod +x /var/lib/gslbd/hc-scripts/check.sh

Configure the health check with scriptPath: /var/lib/gslbd/hc-scripts/check.sh. Scripts survive pod restarts.

Limitation: Each StatefulSet replica has its own PVC. You must copy scripts to every replica (gslbd-0, gslbd-1, …) separately, and new replicas added by scaling start with no scripts. Use this approach for single-replica deployments only.

Option B: ConfigMap (multi-replica, GitOps-friendly)

Declarative, propagates to all replicas automatically including new ones on scale-up. Create the ConfigMap and add it to the manifest (see commented example in deploy/kubernetes/statefulset.yaml):

kubectl create configmap gslbd-hc-scripts \
  --from-file=check.sh=/path/to/your/check.sh \
  -n nexus-gslb

# volumeMount (add alongside existing mounts):
- name: hc-scripts
  mountPath: /etc/gslb/hc-scripts
  readOnly: true

# volume (add alongside existing volumes):
- name: hc-scripts
  configMap:
    name: gslbd-hc-scripts
    defaultMode: 0755   # critical — K8s does not set execute bit by default

Configure with scriptPath: /etc/gslb/hc-scripts/check.sh. 1 MB per ConfigMap limit (scripts should never approach this).

Option C: shared RWX PVC (multi-replica, imperative)

If your cluster has a ReadWriteMany storage class (NFS, CephFS, etc.), a single PVC can be mounted by all replicas simultaneously. Copy scripts once; all pods see them including new replicas:

volumes:
  - name: hc-scripts
    persistentVolumeClaim:
      claimName: gslbd-hc-scripts
      readOnly: false

Option D: scriptPath (VM / bare-metal only)

Set scriptPath to a filesystem path. The script must exist on every node independently. Not portable across container or K8s deployments. Use scriptContent instead unless you have an existing config management system (Ansible, Puppet) already distributing scripts to nodes.

Script contract (all options):

• Receives NEXUS_HC_IP and NEXUS_HC_PORT environment variables
• Exit 0 = healthy; any other exit code = unhealthy
• Killed after timeoutMs milliseconds
• Inherits the gslbd process environment

Webhook checks (recommended for containers)

If managing ConfigMaps or capabilities is undesirable, the webhook type is the container-native alternative to script checks. Run a sidecar or an external service that implements the health logic; gslbd POSTs the target IP and port to it.

# Sidecar example — lightweight health check service on :9191
containers:
  - name: gslbd
    ...
  - name: hc-webhook
    image: mycompany/health-checker:latest
    ports:
      - containerPort: 9191

{
  "type": "webhook",
  "webhookURL": "http://localhost:9191/check",
  "webhookMethod": "POST",
  "intervalMs": 15000,
  "timeoutMs": 3000
}

The webhook service receives: {"ip":"10.0.0.1","port":8080} and returns any 2xx status to indicate health.

HTTP Host / SNI override (http.host)

Most web servers — including Caddy, nginx, and Apache — use SNI-based virtual hosting: the server decides which certificate and site to serve based on the TLS SNI extension in the ClientHello, which is derived from the URL hostname.

Without http.host, health checks connect to the pool member's IP address directly:

https://45.92.9.73/healthz   → Caddy sees SNI = "45.92.9.73" → no matching site → 404 or TLS error

With http.host: "admin.gslb.cc", the TCP dial still goes to the member IP (e.g. 45.92.9.73:443) but the URL presented to the TLS stack — and therefore the SNI value in the ClientHello and the HTTP Host header — is admin.gslb.cc:

TCP connect → 45.92.9.73:443
TLS SNI     → "admin.gslb.cc"    ← Caddy matches this site block
HTTP Host   → "admin.gslb.cc"    ← Caddy checks this for routing

This is essential in any deployment where each node is fronted by a reverse proxy that serves multiple virtual hosts from the same IP.

Example — GSLB-managed service behind Caddy

Pool members: lon-01 (45.92.9.73), eu-01 (65.21.14.204), lab-01 (172.16.1.35). Public FQDN: admin.gslb.cc. Each node runs Caddy with:

admin.gslb.cc {
    reverse_proxy localhost:3000
}

Health check config:

{
  "type": "http",
  "port": 443,
  "httpPath": "/healthz",
  "httpHost": "admin.gslb.cc",
  "tls": true,
  "httpExpectedStatus": 200
}

Nexus probes each member IP (45.92.9.73:443, 65.21.14.204:443, 172.16.1.35:443) while presenting admin.gslb.cc as the SNI and Host. Caddy answers correctly on all three nodes. Without httpHost, probes would fail because Caddy has no site block for bare IPs.

Behavior

• Initial state is optimistic (healthy) until the first probe completes, preventing a brief blackout window at startup.
• Each run, the checker iterates the current endpoint list and updates the in-memory status map atomically per IP.
• The load balancer queries IsHealthy(ip) before returning an endpoint.
• DB write deduplication: UpsertHealthStatus is only called when an endpoint's health state actually changes (unhealthy→healthy or healthy→unhealthy). Stable-state probe results (still healthy, still unhealthy) do not generate DB writes. This eliminates the steady-state write storm in large pools with short check intervals while still persisting every state transition immediately.
• On pool restart (health check updated via API), the dedup cache is cleared so the first post-restart probe always syncs the DB.

Partial health scoring (ScoreWindow)

• Set scoreWindow: N in the health check config to enable a rolling success-rate score for each endpoint.
• N is the window size (number of recent probes). A value of 0 (the default) disables scoring entirely.
• Score: successes / N over the last N probes (0.0–1.0). Before N probes have been recorded the denominator is the actual count of recorded probes.
• IsHealthy gate: an endpoint with scoring enabled is considered healthy as long as its score is > 0 (at least one success in the window). This allows a degraded-but-recoverable backend to continue receiving traffic rather than being hard-cut at the first failure.
• DNS candidate ordering: before the round-robin, WRR, or geo-ip algorithm runs, candidates are sorted descending by score. Higher-scored backends are preferred; score acts as a tiebreaker within the geo-ip preference ordering.
• API exposure: GET /api/v1/pools/{id}/status and GET /api/v1/members/{id}/status include a score field (0.0–1.0) alongside healthy. Persisted in the health_status.score column and served from the DB when no live checker is running.
• Backwards compatibility: ScoreWindow: 0 (the default) leaves all existing behaviour unchanged — binary healthy/unhealthy, no sorting overhead.

Example config with scoring enabled:

health:
  type: http
  port: 443
  checkInterval: 10s
  timeout: 2s
  scoreWindow: 10   # score over last 10 probes; prefer >50%-healthy backends
  http:
    path: "/healthz"
    expectedStatus: 200
    tls: true

Edge cases & timeouts

• Timeouts apply to the TCP dial and to the entire HTTP request via http.Client.Timeout.
• HTTP body is read only as needed for substring matching and capped at 1 MB.
• If an endpoint is removed by GitOps, it is removed from both the checker and the load balancer atomically.
• A log warning is emitted at startup when insecureSkipVerify: true so the setting is never silently active.

Clustered health & co-located backends

When gslbd runs as a multi-node rqlite cluster, be aware of how health interacts with deployments where the GSLB shares addressing with the backends it balances — i.e. a pool member is one of the gslbd nodes itself. This is a common topology (running gslbd on the same boxes as the services).

• Reported health is per-node and rolled up across vantages. The health_status table is keyed by (node_id, pool_id, ip_address): every node persists only its own probe view. The WebUI / API / TUI then aggregate across nodes and report a member as healthy if it is reachable from any node (reachable-from-any-vantage), so a single node that can't reach a backend no longer marks it DOWN cluster-wide. The per-node breakdown is surfaced alongside the rollup — the pool/member status responses include healthyNodes, totalNodes, and a nodes[] array (each node's own healthy/score), the WebUI shows an N/M nodes badge when vantages disagree, and the TUI/gslbctl member view shows the same N/M nodes annotation. DNS routing is still per-node (each node serves what its in-memory checker sees as healthy), so a node will not serve a backend it personally can't reach — but the reported health is no longer corrupted by a single failing vantage. (ARCH-14.)

• Self-checks of a node's own public IPv6 can fail. Many VPS providers route the public IPv6 as a /128 that the host cannot hairpin back to itself (and IPv6 is often disabled on lo). So a node health-checking its own v6 address times out, even though the address is reachable externally. IPv4 usually self-hairpins; IPv6 often does not. Remedy on the affected node (persist both):

  sysctl -w net.ipv6.conf.lo.disable_ipv6=0       # → /etc/sysctl.d/
  ip -6 route add local <node-own-ipv6> dev lo    # → a oneshot systemd unit

• Per-node TLS certs vs a single httpHost. httpHost (SNI / Host header) is set per pool, not per member. If each member node serves its own distinct cert (e.g. node1.example.com, node2.example.com), no single httpHost matches all of them. For a pool of distinct hosts, use insecureSkipVerify: true, a plain tcp check, or a hostname/cert shared across the nodes.

Related

• Automated TLS Certificates (ACME DNS-01) — end-to-end guide for Caddy + GSLB, including the httpHost health check config
• Configuration reference — health.* field definitions

Code references

• internal/health/checker.go: probe implementation, config types, optimistic initial state, httpHost dialer logic.
• internal/health/manager.go: per-pool checker lifecycle, dedup sink (dispatchSink).
• cmd/gslbd/main.go: wiring and lifecycle management.