DNS Query Anomaly Detection (Insights)

License: pro tier and above. See Licensing.

Nexus already has the raw signal — per-service query volume and qtype mix are counted on the hot path regardless. Insights adds a rolling per-hour-of-day baseline on top and alerts through the existing channels (webhook, email, PagerDuty, OpsGenie) when a service's traffic deviates from what's normal for that hour — catching DDoS ramp-up and qtype-flood attacks (ANY/TXT) without a separate analytics product.

v1 is deliberately simple: EWMA mean/variance per metric, no ML, no external time-series database. State is a handful of floats per (service, hour), which replicate via rqlite the same way every other table does.

How It Works

resolved query ──▶ InsightsCollector.Record(domain, qtype)  (cheap, in-memory)
                              │
        every 60s: flush the completed window per service
                              │
   observed metric (this window) vs. this node's own baseline for this hour-of-day
                              │
   |deviation| >= sigmaThreshold  ──▶  alert (cooldown 15 min per service+detector)
                              │
        baseline updated via EWMA regardless (mean, variance, sample count)

Per-node baselines, not merged cluster-wide. Each node persists only its own (service, node, hour, metric) row — the same reasoning as health_status's node_id partitioning: it avoids a concurrent-write race across a multi-node cluster entirely, since no two nodes ever touch the same row. Detection compares a node's own window against its own history, which stays a valid diurnal signal as long as traffic split across nodes is reasonably stable. The WebUI/API average across nodes only for display; the alerting decision itself is always a node comparing its own data.

Detectors (v1)

Detector	Metric	Signature
QPS	`qps`	Spike or drop vs. this hour's baseline — DDoS ramp-up, or an outage cutting off traffic.
Qtype mix	`qtype_any_ratio`, `qtype_txt_ratio`	A sudden shift toward `ANY` or `TXT` queries — a common flood/amplification signature.

Each is independently toggleable (dns.insights.detectors.qps / .qtypeMix).

Deferred, not built

Geo-mix (traffic-hijack symptom via a shift in client geo distribution) needs per-query geo attribution on the hot path that doesn't exist yet.
Unique-qname ratio (the classic "random-subdomain flood" signature) is deferred for an architectural reason, not just laziness: Nexus matches services by exact FQDN, not by zone-apex + wildcard subdomains. A random-subdomain attack against a delegated zone queries qnames that don't match any configured service's exact domain — so a per-service cardinality detector would never observe the traffic it's meant to catch. Building this correctly needs zone-apex attribution (grouping arbitrary subdomains under their parent zone) that this system doesn't have. Rather than ship a detector that can't detect what it's named for, it's deferred alongside geo-mix.

Baseline Math

A metric's baseline is a per-(service, node, hour-of-day) EWMA:

mean_new = mean + α × (observed - mean)
var_new  = (1 - α) × (var + α × (observed - mean)²)

with α = 0.1. The comparison uses a standard-deviation floor (max of the real stddev, 5% of the mean, or 0.01 absolute) so a baseline that's been essentially constant doesn't flag every tiny fluctuation as a multi-sigma anomaly. A baseline needs at least 5 observations before it's trusted enough to alert on — new services (or services with no traffic at a given hour yet) warm up silently first.

The baseline updates via EWMA on every window regardless of whether that window alerted, so a sustained attack can gradually become "the new normal" rather than staying flagged forever. Acceptable for a v1 observability/tiebreaker feature; revisit if that proves a problem in practice.

Configuration

dns:
  insights:
    enabled: true
    sigmaThreshold: 3.0   # standard deviations from baseline before alerting; default 3.0
    detectors:
      qps: true
      qtypeMix: true

Viewing Insights

WebUI

Open a service's detail page — the Insights card shows the baseline mean ± 1σ band by hour-of-day for each enabled metric, with recent anomalies marked, plus a compact list of the last 7 days' anomalies below the chart.

CLI

gslbctl insights <service-id>

Prints the per-node baseline table and the last 7 days of anomalies.

API

curl https://<api-host>:<port>/api/v1/services/<service-id>/insights \
  -H "Authorization: Bearer $TOKEN"

Alerting

Anomalies fire through the same webhook/email/PagerDuty/OpsGenie channels configured under alerts: — no separate configuration needed. Alerts are rate-limited to one per 15 minutes per (service, detector), so a sustained incident produces one alert, not a stream.

Insights observes and alerts only — it does not automatically tighten Response Rate Limiting or take any other corrective action. Leaving that hook for a human (or a future v2) avoids the failure mode of an automated response over-reacting to a false positive.

Notes

Not managed via Terraform — this is diagnostic/observability data, not a declarative resource (same precedent as Config Snapshots and the Query Tracer).
Record() is called for every resolved query regardless of whether it matches a configured service (mirroring the existing QPS tracker), but the number of distinct domains tracked per window is capped at 1000 — rows for domains that aren't real services are simply never read back, so the cap only protects memory/write volume during a flood, it doesn't affect detection for real services.