External Health Signals

License: all tiers.

External health signals let an outside monitor — CloudWatch, Datadog, or any webhook — post a health verdict for a pool member, which Nexus blends with its own health-check probe before deciding whether to answer DNS with that member. This closes the gap where a backend passes Nexus's TCP/HTTP probe but a system Nexus cannot see (a degraded upstream database, a saturated queue, an application-level SLO breach) knows it is unwell.

Signals are ephemeral: each carries a TTL and simply expires. There is no config to persist and no rqlite table — an external verdict with a TTL is the same class of ephemeral health input the post-ARCH contract routes over NATS, alongside heartbeats, RTTs, and RUM.

How It Works

CloudWatch / Datadog / your webhook
        │  POST /api/v1/members/{id}/health-signal   (optionally ?format=<vendor>)
        ▼
   Nexus API ── store.Put(ip, source, healthy, ttl) ──▶ SignalStore (node A)
        │                                                    │
        │                              (clustered) NATS KV mirror
        │                                                    ▼
        │                                    SignalStore on nodes B, C
        ▼
 Manager.IsHealthy(member) = blend(probe result, external verdict, mode)
        ▼
 DNS resolver drops / keeps the member accordingly; self-heals when the TTL lapses

The verdict is consulted at the single Manager.IsHealthy choke point, so every resolution path — normal DNS, the query tracer, filter chains — sees it.

Blend Modes

Each member has an externalSignalMode (WebUI member panel, gslbctl members add/update --signal-mode, Terraform external_signal_mode on nexus_member). It controls how the aggregated external verdict combines with the probe result:

Mode	Member is healthy when…	Use when
`and` (default)	the probe and every external source agree it is healthy	safest — either signal can take a member out
`or`	the probe or an external source reports healthy	external monitor can keep a member in despite a probe blip
`override`	the external verdict wins outright (probe ignored)	the external system is the source of truth

Multiple sources per member aggregate conservatively: any unexpired source reporting unhealthy makes the member externally unhealthy, so any monitor flagging degradation counts.

Ingesting a Verdict

POST /api/v1/members/{id}/health-signal (permission members:write). Native body:

{ "source": "datadog", "healthy": false, "score": 0.2, "ttl": 120 }

source (required) — a stable label; verdicts from the same source overwrite.
healthy (required) — the verdict.
score — optional 0–1 health score (informational).
ttl — seconds the verdict stays live (default 300, max 3600). Post again before it lapses to keep it, or post healthy:true / let it expire to clear it.

Vendor adapters

Most monitors will not emit the native shape, so ?format=<vendor> translates a vendor payload in-endpoint (no separate adapter service):

`format`	Payload	Healthy when	Source label
`cloudwatch`	`{"AlarmName","NewStateValue"}`	`NewStateValue == "OK"`	`cloudwatch:<AlarmName>`
`datadog`	`{"alert_transition","alert_type","title"}`	`Recovered` / `alert_type == "success"`	`datadog:<title>`
`generic`	`{"healthy","score?","source?","ttl?"}`	`healthy` field	`source` or `webhook`

CloudWatch: deliver the SNS message with raw message delivery (or unwrap it upstream) so the endpoint sees the alarm JSON directly. Datadog: configure the webhook body as {"alert_transition":"$ALERT_TRANSITION","alert_type":"$ALERT_TYPE","title":"$EVENT_TITLE"}.

Reading Live Verdicts

GET /api/v1/members/{id}/health-signals (permission members:read) returns the member's unexpired signals with their source, verdict, score, and expiry. The WebUI member panel shows the same list live, and the topology member node carries an ext:<mode> pill when a non-default blend mode governs the member.

Cluster Propagation

A verdict is posted to a single node's API. When NATS state sync is configured, that node mirrors the signal into the gslb_<cluster>_health_signals KV bucket (2h TTL); every node watches the bucket and applies peer verdicts into its own store, so all nodes route around the member consistently. Peer-applied verdicts are not re-published (no loop). Without NATS, signals stay node-local — the receiving node blends them and still works, but other nodes are unaffected until they receive their own verdict.

The KV bucket-create failure degrades to node-local; it never blocks startup.

Notes

Verdicts survive a gslbd restart only as long as the poster keeps refreshing them within the TTL — there is no durable state by design.
The per-member mode is read from a 30s-cached lookup, so a mode change can take up to 30s to affect blending.
OIDC has no bearing here; ingestion is gated purely by members:write.