Operations & Runbooks

This guide covers day-2 operations: monitoring, troubleshooting, routine tasks, and failure handling.

Monitoring

• Prometheus metrics (see docs/Metrics.md). Suggested dashboards:
  • DNS throughput: gslbd_dns_requests_total rate.
  • Health status: gslbd_health_endpoints_healthy{family}.
  • NATS connectivity: gslbd_state_nats_connected.
  • Merge lag p95: histogram quantiles on gslbd_state_merge_lag_ms.
  • Active members: gslbd_state_active_members.

Logs

• GitOps reconcile logs: fetch/verify/apply outcomes and errors.
• NATS errors: connection/disconnect and subscriber drain errors.
• DNS errors: server start/stop issues.

Routine tasks

• Rotate GPG keys: update trust store on nodes; update allowedSigners list; merge a new signed commit to test.
• Rotate NATS creds/certs: generate new creds; restart gslbd or trigger reconnect.
• Update endpoints: commit to Git repo; watcher applies within pollInterval.

Troubleshooting

!!! tip "Start with Metrics" When troubleshooting, check the Prometheus metrics first — they often reveal the root cause faster than logs.

• No DNS answers:
  • Check licensing limit: look for SERVFAIL and rate limit logs.
  • Verify endpoints configured and healthy (gslbd_health_endpoints_healthy).
  • Confirm domain matches dns.domain and query suffix is exact (trailing dot).
• GitOps changes not applied:
  • Verify signatures (gslbd_gitops_verify_total{result="error"}) and signer allowlist.
  • Check validation errors and logs.
  • Ensure pathPrefix and repo URL are correct; test git access on host.
• State sync not working:
  • Check gslbd_state_nats_connected.
  • Verify NATS URLs and TLS/auth settings; test nats CLI.
  • Confirm JetStream enabled and KV buckets created.
• Global quorum policy seems off:
  • Ensure heartbeatInterval/heartbeatTTL reflect reality; active membership must be non-zero.
  • Validate quorumMinPercent and that enough nodes are reporting within TTL.

Maintenance windows

• Use GitOps to stage config changes with signed commits.
• Consider "freeze windows" by temporarily pausing GitOps changes (set repoURL empty or pause at network layer).

Testing WAN partitions

• Use tc/netem to inject latency/loss; observe merge lag and policy behavior.
• Ensure local-only fallback serves queries when NATS is unreachable.

Cluster status

Runtime NATS connectivity state is available via the API (GET /api/v1/cluster/status) and the TUI. It reports whether this node is connected to NATS, the server it is connected to, and the current round-trip time.

Terraform

The nexus_cluster_status data source reads live cluster connectivity for use in Terraform plans and outputs.

data "nexus_cluster_status" "this" {}

output "nats_connected" {
  value = data.nexus_cluster_status.this.connected
}

output "nats_rtt_ms" {
  value = data.nexus_cluster_status.this.rtt_ms
}

Attribute	Type	Description
cluster_id	string	Cluster identifier.
cluster_name	string	NATS cluster name.
connected	bool	Whether the node is connected to NATS.
nats_enabled	bool	Whether NATS is enabled on this node.
rtt_ms	number	Round-trip time to the NATS server in milliseconds.
server_name	string	Connected NATS server name.
server_url	string	Connected NATS server URL.