Metrics (Prometheus)

Nexus GSLB exposes Prometheus-compatible metrics at a configurable HTTP endpoint when enabled. All metrics include constant labels cluster and node when these are set in the config.

Enable endpoint

metrics:
  enablePrometheus: true
  listenAddr: "0.0.0.0"
  port: 9090

• Metrics are available at http://<listenAddr>:<port>/metrics.

Metric catalog (namespace: gslbd)

• dns
  • gslbd_dns_requests_total
  • gslbd_dns_query_duration_seconds (histogram) — end-to-end DNS handler latency; target p99 < 10 ms
• gitops
  • gslbd_gitops_fetch_total{result}
  • gslbd_gitops_verify_total{result}
  • gslbd_gitops_apply_total{result}
  • gslbd_gitops_last_apply_info{sha,signer} value 1 for the last applied commit
• state (NATS/JetStream)
  • gslbd_state_nats_connected (0/1)
  • gslbd_state_nats_published_total{type}
  • gslbd_state_nats_received_total{type}
  • gslbd_state_kv_put_total{bucket,result}
  • gslbd_state_kv_get_total{bucket,result}
  • gslbd_state_merge_lag_ms (histogram) — time between event timestamp and peer receipt; measures NATS transit lag
  • gslbd_state_active_members (gauge)
• health
  • gslbd_health_endpoints_total{family}
  • gslbd_health_endpoints_healthy{family}
  • gslbd_pool_members_total{pool_id} (gauge) — total tracked endpoints per pool
  • gslbd_pool_members_healthy{pool_id} (gauge) — healthy endpoints per pool (last probe)
  • gslbd_health_failover_detection_seconds{direction} (histogram) — Layer 1 failover detection latency: time from the first probe that disagreed with the confirmed state to the state-change callback. direction is down (healthy→unhealthy) or up (unhealthy→healthy). With no scoreWindow, this is near-zero (single probe). With a scoreWindow, this measures the multi-probe confirmation delay.
• gitops (emitted only when a GitOps repo is configured)
  • gslbd_gitops_fetch_total{result} — repo fetch attempts (success/error)
  • gslbd_gitops_verify_total{result} — commit signature verification attempts
  • gslbd_gitops_apply_total{result} — config apply (write + restart) attempts
  • gslbd_gitops_last_apply_info{sha,signer} (gauge) — last successfully applied commit
• dnssec (emitted only when dnssec.enabled: true)
  • gslbd_dnssec_sign_duration_seconds (histogram) — per-RRset signing latency; target p99 < 1 ms
  • gslbd_dnssec_key_days_remaining{type} (gauge) — days until KSK/ZSK expiry; label value is ksk or zsk
  • gslbd_dnssec_response_bytes (histogram) — signed response size in bytes; responses > 1232 bytes trigger TC=1 truncation

Scrape example

scrape_configs:
  - job_name: 'gslbd'
    scrape_interval: 15s
    static_configs:
      - targets: ['gslbd-hostname:9090']

Sample alerts

groups:
- name: gslbd
  rules:
  - alert: GslbdNATSDisconnected
    expr: gslbd_state_nats_connected == 0
    for: 2m
  - alert: GslbdMergeLagHigh
    expr: histogram_quantile(0.95, sum(rate(gslbd_state_merge_lag_ms_bucket[5m])) by (le)) > 2000
    for: 5m
  - alert: GslbdActiveMembersZero
    expr: gslbd_state_active_members == 0
    for: 5m
  - alert: GslbdPoolDegraded
    # Fires when any pool has fewer than 1 healthy member (DNS would return no records).
    expr: gslbd_pool_members_healthy < 1
    for: 30s
    labels:
      severity: critical
  - alert: GslbdFailoverDetectionSlow
    # Fires when p95 detection latency exceeds 2× the expected check interval.
    # Adjust threshold to match your checkInterval setting.
    expr: histogram_quantile(0.95, sum(rate(gslbd_health_failover_detection_seconds_bucket[15m])) by (le, direction)) > 20
    for: 10m
    labels:
      severity: warning
  - alert: GslbdDNSSECKeyExpiryWarning
    expr: gslbd_dnssec_key_days_remaining < 30
    for: 1h
    labels:
      severity: warning
  - alert: GslbdDNSSECKeyExpiryCritical
    expr: gslbd_dnssec_key_days_remaining < 7
    for: 1h
    labels:
      severity: critical

Implementation references

• internal/metrics/* for registerers, collectors, and HTTP server.
• Metrics are registered with const labels via metrics.InitLabels(clusterID, nodeID) in main.

See Telemetry.md for OpenTelemetry distributed tracing coverage.

Metrics with other backends (InfluxDB, Elastic/Kibana, Datadog, ...)

Nexus GSLB doesn't ship a native writer for every time-series backend — instead it exposes standard Prometheus scrape (/metrics) and OTLP traces (see Telemetry Reference), and lets your existing collector agent bridge from there. This avoids a custom exporter (and its own maintenance burden) per backend.

InfluxDB

Use Telegraf's inputs.prometheus plugin to scrape /metrics and write to InfluxDB:

# telegraf.conf
[[inputs.prometheus]]
  urls = ["http://gslbd-hostname:9090/metrics"]

[[outputs.influxdb_v2]]
  urls = ["http://influxdb:8086"]
  token = "$INFLUX_TOKEN"
  organization = "your-org"
  bucket = "gslbd"

InfluxDB's own scraper (influxdb.scrape) can also pull the endpoint directly without Telegraf, if you're on a version that supports Prometheus-format scrape targets.

Elastic / Kibana

Two options, depending on what's already in your stack:

• Elastic Agent — add the Prometheus input integration, pointed at http://gslbd-hostname:9090/metrics. Metrics land in the metrics-prometheus.* data stream and are visible in Kibana out of the box.
• OTel Collector → Elastic — since Elastic accepts OTLP natively for Observability/APM, point the OTel Collector's otlp exporter at your Elastic endpoint. Useful if you're already running a Collector to bridge traces and want metrics on the same pipe.

General pattern

Any backend that can either scrape a Prometheus endpoint or receive OTLP can be wired up the same way — via an existing agent/collector, not custom code in gslbd. The OTel Collector in particular can fan a single OTLP stream out to Jaeger, Zipkin, Prometheus remote-write, Elastic, and more simultaneously (see Telemetry Reference for the collector config).