Encrypted DNS (DoT/DoH)

Encrypted DNS: DoT and DoH

Nexus GSLB supports two encrypted DNS transports:

Protocol	RFC	Default port	Use case
DNS over TLS (DoT)	RFC 7858	853	Stub resolvers, mobile OS, enterprise DNS policies
DNS over HTTPS (DoH)	RFC 8484	443	Browsers, applications, HTTP proxies

Both transports share the same resolution logic as plain UDP/53. DNSSEC signing works identically across all transports. Unlike UDP, TCP-based transports never need to truncate large responses — Nexus detects the transport automatically and skips the 1232-byte UDP payload limit for DoT and DoH connections.

Prerequisites

Both DoT and DoH require a valid TLS certificate and private key. The certificate must cover the hostname your clients will connect to.

Option A — Let's Encrypt (recommended for public deployments)

certbot certonly --standalone -d ns1.example.com
# Certificate: /etc/letsencrypt/live/ns1.example.com/fullchain.pem
# Key:         /etc/letsencrypt/live/ns1.example.com/privkey.pem

Option B — Self-signed (internal/lab use only)

openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
  -keyout /etc/nexus/dot.key -out /etc/nexus/dot.crt \
  -days 365 -nodes -subj "/CN=ns1.example.com"

Clients querying a self-signed certificate will need to trust it explicitly (e.g. via a CA bundle or system trust store).

Configuration

Both listeners are disabled by default. Add the relevant block to your config.yaml under the dns: section.

DNS over TLS (DoT)

dns:
  listenAddr: "0.0.0.0"
  port: 53
  domain: gslb.example.com

  dot:
    enabled: true
    port: 853               # optional; 853 is the default
    certFile: /etc/nexus/tls/fullchain.pem
    keyFile:  /etc/nexus/tls/privkey.pem

DNS over HTTPS (DoH)

dns:
  listenAddr: "0.0.0.0"
  port: 53
  domain: gslb.example.com

  doh:
    enabled: true
    port: 443               # optional; 443 is the default
    certFile: /etc/nexus/tls/fullchain.pem
    keyFile:  /etc/nexus/tls/privkey.pem
    path: /dns-query        # optional; /dns-query is the default

Both enabled simultaneously

DoT and DoH can run at the same time alongside plain UDP/53. All three listeners share the same certificate files if you want to reuse a single cert:

dns:
  listenAddr: "0.0.0.0"
  port: 53
  domain: gslb.example.com
  dot:
    enabled: true
    certFile: /etc/nexus/tls/fullchain.pem
    keyFile:  /etc/nexus/tls/privkey.pem
  doh:
    enabled: true
    certFile: /etc/nexus/tls/fullchain.pem
    keyFile:  /etc/nexus/tls/privkey.pem

Configuration reference

Field	Type	Default	Description
`dns.dot.enabled`	bool	`false`	Enable DoT listener
`dns.dot.port`	int	`853`	TCP port for DoT
`dns.dot.certFile`	string	—	Path to PEM certificate (required when enabled)
`dns.dot.keyFile`	string	—	Path to PEM private key (required when enabled)
`dns.doh.enabled`	bool	`false`	Enable DoH listener
`dns.doh.port`	int	`443`	TCP port for DoH
`dns.doh.certFile`	string	—	Path to PEM certificate (required when enabled)
`dns.doh.keyFile`	string	—	Path to PEM private key (required when enabled)
`dns.doh.path`	string	`/dns-query`	HTTP path for the DoH endpoint

Firewall rules

Open the new ports in addition to the existing UDP/53 rule:

# DoT
ufw allow 853/tcp comment "DNS over TLS"

# DoH (if not sharing port 443 with the web UI)
ufw allow 443/tcp comment "DNS over HTTPS / HTTPS"

If the Nexus web UI and the DoH listener both need port 443 on the same address, put a reverse proxy (e.g. Caddy or nginx) in front and forward /dns-query to a non-standard DoH port (e.g. 8443):

dns:
  doh:
    enabled: true
    port: 8443
    path: /dns-query
    certFile: /etc/nexus/tls/fullchain.pem
    keyFile:  /etc/nexus/tls/privkey.pem

# Caddyfile excerpt
ns1.example.com {
    reverse_proxy /dns-query localhost:8443 {
        transport http { tls_insecure_skip_verify }
    }
}

Testing

DoT with `kdig` (from knot-dnsutils)

kdig @ns1.example.com +tls app.gslb.example.com A
# Self-signed cert:
kdig @ns1.example.com +tls +tls-ca=/etc/nexus/tls/ca.crt app.gslb.example.com A

DoH with `curl`

# POST (wire format)
DNS_MSG=$(printf '\x00\x01\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x03app\x04gslb\x07example\x03com\x00\x00\x01\x00\x01')
curl -s -X POST "https://ns1.example.com/dns-query" \
  -H "Content-Type: application/dns-message" \
  --data-binary "$DNS_MSG" \
  --output - | hexdump -C

# GET (base64url-encoded query — use dog or doh-client for convenience)
dog app.gslb.example.com A --tls-host ns1.example.com

DoH with `dog`

# Install: https://github.com/ogham/dog
dog @https://ns1.example.com/dns-query app.gslb.example.com A

Verify DNSSEC over DoH

kdig @ns1.example.com +https=/dns-query +dnssec app.gslb.example.com A
# Expect: RRSIG record in the answer section, AD flag set

Certificate reload

Nexus loads the TLS certificate once at startup. To roll a renewed certificate without downtime:

Replace the cert/key files on disk (e.g. via a certbot deploy hook or cp).
Send SIGHUP or use the API restart endpoint:

curl -X POST https://nexus-api.example.com/api/v1/restart \
  -H "Authorization: Bearer $GSLB_API_KEY"

The daemon performs a graceful restart — in-flight DNS queries complete before the listener is torn down. Certbot deploy hook example:

# /etc/letsencrypt/renewal-hooks/deploy/nexus-reload.sh
#!/bin/bash
cp /etc/letsencrypt/live/ns1.example.com/fullchain.pem /etc/nexus/tls/fullchain.pem
cp /etc/letsencrypt/live/ns1.example.com/privkey.pem  /etc/nexus/tls/privkey.pem
systemctl reload-or-restart gslbd

Performance notes

DoT and DoH have higher per-query overhead than UDP/53 due to the TLS handshake. Modern clients reuse persistent connections, so the handshake cost is amortised over many queries.
GSLB responses are typically small (a single A or AAAA record) and will be served from the in-memory service cache without the TCP payload size limit ever being relevant.
If you are serving many short-lived clients (e.g. IoT devices that do not reuse connections), consider placing a caching DoH resolver (e.g. Unbound, Pi-hole) in front of Nexus and pointing it at UDP/53 for upstream queries.

Client configuration examples

Android (Private DNS)

Settings → Network & Internet → Private DNS → enter ns1.example.com (uses DoT on port 853 automatically).

iOS / macOS (DNS over HTTPS via profile)

Create a .mobileconfig profile with:

<key>DNSSettings</key>
<dict>
  <key>DNSProtocol</key>
  <string>HTTPS</string>
  <key>ServerURL</key>
  <string>https://ns1.example.com/dns-query</string>
</dict>

Deploy via Apple Configurator or MDM.

Firefox

about:config → set network.trr.mode to 2 and network.trr.uri to https://ns1.example.com/dns-query.

systemd-resolved (Linux)

# /etc/systemd/resolved.conf
[Resolve]
DNS=ns1.example.com
DNSOverTLS=yes

systemctl restart systemd-resolved
resolvectl status

Unbound (forwarding resolver)

forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-addr: <nexus-ip>@853#ns1.example.com