Health Alerting

Nexus GSLB can notify your team when an endpoint becomes unhealthy or recovers. Alerts fire on state transitions only — you get one notification when something breaks, and one when it recovers. You do not get a flood of repeated alerts while an endpoint stays down.

Four delivery targets are supported and can be used simultaneously:

License: SMTP and webhook channels require the paid tier or above. PagerDuty and Opsgenie channels require the pro tier or above. Contact licensing@gslb.nexus to upgrade.

Two ways to configure alert channels

1. Config file (gslbd.yaml) — static channels defined at startup. Best for infrastructure-as-code deployments where channels rarely change.

2. API / WebUI / TUI — dynamic channels stored in the database. Hot-reloaded on create/update/delete without a restart. Best for multi-tenant deployments or when you want to manage channels via the UI.

Both sources are merged at startup. Config-file channels are never overwritten by the API.

What is PagerDuty?

PagerDuty is a cloud service that wakes up the right on-call engineer when something breaks. You connect Nexus to a PagerDuty "service" (a logical grouping of alerts). When Nexus reports an unhealthy endpoint, PagerDuty opens an incident and escalates it according to your on-call schedule — SMS, phone call, Slack, email, and mobile push are all possible. When the endpoint recovers, Nexus tells PagerDuty to resolve the incident automatically, so no manual cleanup is needed.

What is OpsGenie?

OpsGenie (now part of Atlassian) does the same job as PagerDuty. It receives an alert from Nexus when an endpoint goes down, routes it to whoever is on call, and closes the alert automatically when the endpoint recovers. If you already use Jira or Confluence, OpsGenie integrates tightly with those tools.

SMTP configuration

SMTP can be configured two ways. The database (WebUI/API) takes precedence; YAML applies when no DB config exists.

WebUI (recommended)

Go to Admin → Settings → SMTP / Email. Changes take effect immediately without a restart. The "Test" button verifies delivery before saving. See SMTP Settings for full details.

Config-file configuration

All alert targets live under the alerts: key in gslbd.yaml. You can configure any combination — they all fire for the same health transitions.

SMTP (email via YAML)

Nexus can send email directly via SMTP. Both STARTTLS (port 587) and implicit TLS (port 465 "SMTPS") are supported.

alerts:
  smtp:
    - host: "smtp.example.com"
      port: 587
      from: "nexus-gslb@example.com"
      to:
        - "ops@example.com"
        - "oncall@example.com"
      username: "nexus-gslb@example.com"
      password: "app-password-here"
      startTLS: true    # use STARTTLS on port 587 (default false)
      tls: false        # use implicit TLS on port 465 (default false)

Example email subject and body:

Subject: Nexus GSLB: web-prod member 203.0.113.10 is UNHEALTHY

Pool:     web-prod
Endpoint: 203.0.113.10
Status:   UNHEALTHY
Time:     2026-05-22 14:30:00 UTC

Generic webhook

A webhook is a plain HTTP POST to any URL. Nexus sends a JSON body and optionally signs it with HMAC-SHA256 so the receiving server can verify the request came from Nexus.

alerts:
  webhooks:
    - url: "https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
      secret: ""          # leave empty if you don't need signature verification
    - url: "https://automation.internal/nexus-alert"
      secret: "my-signing-secret"   # the receiver checks X-Nexus-Signature

Payload (sent on every transition):

{
  "poolId":    "web-prod",
  "ipAddress": "203.0.113.10",
  "healthy":   false,
  "timestamp": 1716000000
}

Signature verification (when secret is set):

Nexus sends an X-Nexus-Signature: sha256=<hex> header. Verify it on the receiving end:

import hmac, hashlib
expected = "sha256=" + hmac.new(secret.encode(), body, hashlib.sha256).hexdigest()
assert header == expected

mac := hmac.New(sha256.New, []byte(secret))
mac.Write(body)
expected := "sha256=" + hex.EncodeToString(mac.Sum(nil))
// compare with r.Header.Get("X-Nexus-Signature")

PagerDuty

Step 1 — Create an integration in PagerDuty

1. Log into PagerDuty → Services → select the service that should own GSLB alerts (or create a new one).
2. Click the Integrations tab → Add an Integration.
3. Choose Events API v2.
4. Copy the Integration Key that appears — this is your routingKey.

Step 2 — Add to gslbd.yaml

alerts:
  pagerduty:
    - routingKey: "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"
      severity: "critical"    # optional; default "critical"

severity controls how PagerDuty classifies the incident. Choose based on how urgently you want to be paged:

What Nexus sends to PagerDuty:

When an endpoint goes unhealthy:

{
  "routing_key":  "a1b2c3d4...",
  "event_action": "trigger",
  "dedup_key":    "nexus/web-prod/203.0.113.10",
  "payload": {
    "summary":   "Pool web-prod: endpoint 203.0.113.10 is unhealthy",
    "severity":  "critical",
    "source":    "nexus-gslb",
    "timestamp": "2026-05-17T14:30:00Z",
    "custom_details": {
      "pool_id":  "web-prod",
      "endpoint": "203.0.113.10"
    }
  }
}

When the endpoint recovers, Nexus sends "event_action": "resolve" with the same dedup_key. PagerDuty uses the dedup_key to match the resolve to the original incident and close it automatically.

Multiple PagerDuty services:

You can route alerts to different PagerDuty services for different teams:

alerts:
  pagerduty:
    - routingKey: "key-for-networking-team"
      severity: "critical"
    - routingKey: "key-for-management-dashboard"
      severity: "warning"

OpsGenie

Step 1 — Create an API key in OpsGenie

1. Log into OpsGenie → Settings → API key management → Add new API key.
2. Give it a name (e.g. "Nexus GSLB").
3. Grant it Create and Update and Close alert permissions.
4. Copy the generated key — this is your apiKey.

Step 2 — Add to gslbd.yaml

alerts:
  opsgenie:
    - apiKey: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
      priority: "P1"    # optional; default "P1"
      region: "us"      # optional; "us" (default) or "eu"

priority maps to OpsGenie's P1–P5 scale:

region selects the API endpoint. Use "eu" only if your OpsGenie account is on the EU data residency plan (check under Settings → Account).

What Nexus sends to OpsGenie:

When an endpoint goes unhealthy, Nexus POSTs to https://api.opsgenie.com/v2/alerts:

{
  "message":     "Pool web-prod: endpoint 203.0.113.10 is unhealthy",
  "alias":       "nexus/web-prod/203.0.113.10",
  "description": "Nexus GSLB detected that endpoint 203.0.113.10 in pool web-prod is unhealthy...",
  "priority":    "P1",
  "details": {
    "pool_id":  "web-prod",
    "endpoint": "203.0.113.10"
  }
}

When the endpoint recovers, Nexus sends a close request to /v2/alerts/nexus%2Fweb-prod%2F203.0.113.10/close?identifierType=alias. OpsGenie uses the alias to match the close to the original alert.

Managing channels via the API / WebUI / TUI

Alert channels can be created, updated, and deleted at runtime without restarting the daemon. Changes take effect immediately — no config file edit required.

WebUI

Navigate to Admin → Alerting. From there you can:

• Create a new channel (select type, fill in the type-specific form)
• Enable or disable any channel
• Send a test alert to verify delivery
• Delete channels you no longer need

API

# List channels
curl -H "Authorization: Bearer $TOKEN" https://gslb.example.com:8880/api/v1/alerting/channels

# Create an SMTP channel
curl -X POST -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Ops email",
    "type": "smtp",
    "config": {
      "host": "smtp.example.com",
      "port": 587,
      "from": "nexus@example.com",
      "to": ["ops@example.com"],
      "username": "nexus@example.com",
      "password": "secret",
      "startTLS": true
    },
    "enabled": true
  }' \
  https://gslb.example.com:8880/api/v1/alerting/channels

# Send a test alert
curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://gslb.example.com:8880/api/v1/alerting/channels/{id}/test

# Update a channel (send "***" for a secret field to leave it unchanged)
curl -X PUT -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "Ops email", "config": {"password": "***", ...}, "enabled": true}' \
  https://gslb.example.com:8880/api/v1/alerting/channels/{id}

# Delete a channel
curl -X DELETE -H "Authorization: Bearer $TOKEN" \
  https://gslb.example.com:8880/api/v1/alerting/channels/{id}

Note: GET responses mask sensitive fields (passwords, API keys, routing keys) with "***". When updating a channel, send "***" for any secret you don't want to change — the server will preserve the existing stored value.

TUI (gslbctl)

Press A from the main pools screen to open the Alerting screen. From there:

• ↑/↓ — navigate the channel list
• n — create a new channel (interactive form)
• d — delete the selected channel
• Esc — go back

Terraform

resource "nexus_alert_channel" "ops_email" {
  name    = "Ops email"
  type    = "smtp"
  enabled = true
  config  = jsonencode({
    host     = "smtp.example.com"
    port     = 587
    from     = "nexus@example.com"
    to       = ["ops@example.com"]
    username = "nexus@example.com"
    password = var.smtp_password
    startTLS = true
  })
}

resource "nexus_alert_channel" "pagerduty_prod" {
  name    = "PagerDuty prod"
  type    = "pagerduty"
  enabled = true
  config  = jsonencode({
    RoutingKey = var.pagerduty_routing_key
    Severity   = "critical"
  })
}

Using all four together

All targets fire for the same health transition. A typical enterprise setup pages on-call engineers via PagerDuty while also posting to a Slack channel via webhook and emailing the ops list:

alerts:
  smtp:
    - host: "smtp.example.com"
      port: 587
      from: "nexus@example.com"
      to: ["ops@example.com"]
      username: "nexus@example.com"
      password: "secret"
      startTLS: true
  webhooks:
    - url: "https://hooks.slack.com/services/..."
  pagerduty:
    - routingKey: "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"
      severity: "critical"

Testing your configuration

All channel types — use the built-in test endpoint to fire a synthetic alert through any DB-managed channel:

curl -X POST -H "Authorization: Bearer $TOKEN" \
  https://gslb.example.com:8880/api/v1/alerting/channels/{id}/test

The test dispatches a synthetic UNHEALTHY event for a fake pool and endpoint. This is the quickest way to verify connectivity, credentials, and routing without waiting for a real health event. The WebUI Alerting page also has a "Test" button for each channel.

Webhook — use curl to post a sample payload and confirm your receiver handles it:

curl -X POST https://your-endpoint/nexus-alert \
  -H "Content-Type: application/json" \
  -d '{"poolId":"test","ipAddress":"1.2.3.4","healthy":false,"timestamp":1716000000}'

PagerDuty — send a test event via the PagerDuty API directly:

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{
    "routing_key": "YOUR_ROUTING_KEY",
    "event_action": "trigger",
    "dedup_key": "nexus-test",
    "payload": {
      "summary": "Nexus GSLB test alert",
      "severity": "info",
      "source": "nexus-gslb"
    }
  }'

Then resolve it:

curl -X POST https://events.pagerduty.com/v2/enqueue \
  -H "Content-Type: application/json" \
  -d '{"routing_key":"YOUR_ROUTING_KEY","event_action":"resolve","dedup_key":"nexus-test"}'

OpsGenie — create and immediately close a test alert:

curl -X POST https://api.opsgenie.com/v2/alerts \
  -H "Content-Type: application/json" \
  -H "Authorization: GenieKey YOUR_API_KEY" \
  -d '{"message":"Nexus GSLB test","alias":"nexus-test","priority":"P5"}'

curl -X POST "https://api.opsgenie.com/v2/alerts/nexus-test/close?identifierType=alias" \
  -H "Content-Type: application/json" \
  -H "Authorization: GenieKey YOUR_API_KEY" \
  -d '{"note":"Test complete"}'

Delivery behaviour

• Retry: each alert attempt is retried up to 3 times with exponential backoff (2 s, 4 s).
• Non-blocking: the DNS query path is never delayed by alerting. Events are queued in a 256-slot buffer and dispatched by a background goroutine.
• Transitions only: if an endpoint stays unhealthy, no repeated alerts are sent. A new alert fires only when the state changes.
• Buffer overflow: if all 256 slots are full (e.g., a target is unreachable and retries are stacking up), new events are dropped and logged at WARN level. This is a safety valve to prevent memory growth during a prolonged outage of the alerting backend.

Troubleshooting

No alerts received

1. Check daemon logs for alerter: delivery permanently failed — this indicates the target URL is unreachable or returning errors.
2. Confirm the routing key / API key is correct and has not been revoked.
3. Verify the endpoint is actually transitioning health state — alerts only fire on transitions. If the endpoint was already unhealthy when the daemon started, the first probe result is treated as a transition.

PagerDuty shows "invalid routing key"

The integration key is per-service in PagerDuty, not per-account. Confirm you copied the key from Services → your service → Integrations, not from account settings.

OpsGenie returns 403

The API key does not have the correct permissions. Edit the key under Settings → API key management and ensure Create and Update and Close are both checked.

OpsGenie alerts not closing

Nexus closes alerts using the alias field (nexus/<poolId>/<ipAddress>). If the alias was changed or an alert was created without this alias (e.g., manually), the close request will 404. This is logged at WARN level and does not affect daemon operation.

Alerts firing on every daemon restart

On startup, Nexus treats the first health probe result for each endpoint as a transition. If you restart the daemon frequently, you may see spurious alerts for already-healthy endpoints. This is by design — the daemon has no memory of previous state across restarts. Set a short health check interval (e.g., interval: "10s") so endpoints reach a stable state quickly after restart.