BGP Route Health Injection

BGP Route Health Injection Setup

This guide walks through configuring Nexus GSLB to inject and withdraw BGP routes based on health check results. It covers three common deployment scenarios, peer configuration examples for FRR (Linux) and Cisco IOS, and troubleshooting steps.

For the full field reference see BGP Configuration Reference.

Concepts

Why BGP RHI alongside DNS GSLB?

DNS-based load balancing is bounded by TTL: clients that cached a response before a failure keep hitting the dead endpoint until the TTL expires. BGP RHI operates at the network layer — when a prefix is withdrawn, the upstream router immediately stops forwarding traffic to that GSLB node, regardless of what DNS clients cached. The two mechanisms complement each other:

Layer	Mechanism	Failover speed
Network	BGP prefix withdrawal	Seconds (BGP convergence)
DNS	TTL expiry + new answer	Minutes (TTL dependent)

Node prefixes vs pool prefixes

Node prefixes are announced as long as the daemon is alive. Use them for the IP address of the GSLB service itself in an anycast setup — multiple nodes advertise the same prefix, BGP ECMP distributes DNS queries across them, and a failed node withdraws so its traffic shifts to survivors.
Pool prefixes are tied to the health of a specific pool's members. Use them when your backend application servers run on an anycast or shared IP range and you want to withdraw a route when too many backends are down.

Scenario 1 — Single node, anycast VIP for the GSLB service

The simplest case: one Nexus node announces a VIP for its DNS service. If the daemon stops, the VIP disappears and clients fall through to another resolver.

Network topology:

Internet clients
      │  DNS queries to 203.0.113.1
      ▼
  Router (ASN 65000)
  10.0.0.254
      │ BGP session
      ▼
  Nexus node (ASN 65001)
  10.0.0.1  ──announces──▶  203.0.113.0/24

gslbd.yaml:

bgp:
  enabled: true
  localASN: 65001
  routerID: "10.0.0.1"
  listenPort: 179
  holdTime: "90s"
  keepAliveTime: "30s"
  peers:
    - remoteASN: 65000
      remoteAddr: "10.0.0.254"
  nodePrefixes:
    - "203.0.113.0/24"

The daemon announces 203.0.113.0/24 immediately after the BGP session establishes. On systemctl stop gslbd or kill -TERM, GoBGP sends a BGP NOTIFICATION to the peer before exit, which causes the peer to withdraw the route within seconds.

Scenario 2 — Three-node anycast cluster (ECMP)

Three Nexus nodes each announce the same prefix. The upstream router load-balances DNS queries across all three via ECMP. If one node fails, it withdraws its advertisement and the remaining two absorb its traffic.

Network topology:

                 Router (ASN 65000)
                 ┌────────────────┐
                 │   ECMP table   │
                 │ 203.0.113.0/24 │◄── lon-01 (10.100.0.1)
                 │ 203.0.113.0/24 │◄── eu-01  (10.100.0.2)
                 │ 203.0.113.0/24 │◄── lab-01 (10.100.0.3)
                 └────────────────┘

Each node has an identical bgp: block except for routerID, which must be unique per node.

gslbd.yaml on lon-01:

node:
  id: "lon-01"

bgp:
  enabled: true
  localASN: 65001
  routerID: "10.100.0.1"      # unique per node
  listenPort: 179
  holdTime: "90s"
  keepAliveTime: "30s"
  peers:
    - remoteASN: 65000
      remoteAddr: "10.100.0.254"   # upstream router
  nodePrefixes:
    - "203.0.113.0/24"             # shared anycast prefix

gslbd.yaml on eu-01:

node:
  id: "eu-01"

bgp:
  enabled: true
  localASN: 65001
  routerID: "10.100.0.2"
  listenPort: 179
  holdTime: "90s"
  keepAliveTime: "30s"
  peers:
    - remoteASN: 65000
      remoteAddr: "10.100.0.254"
  nodePrefixes:
    - "203.0.113.0/24"

!!! tip "Router ID must be unique" Even though all nodes share the same ASN (localASN: 65001) for iBGP designs, the routerID must differ per node. Using the node's management or loopback IP is the standard approach.

!!! warning "ECMP hash and sticky sessions" ECMP distributes flows by 5-tuple hash. DNS over UDP is stateless so this is fine. If you run the Nexus API or HTTPS over the anycast VIP, TCP sessions may hash differently after a node failure — consider session affinity at the application layer if needed.

Scenario 3 — Pool-driven prefix injection

Backend application servers share an anycast IP range. Nexus injects the prefix when the pool is healthy and withdraws it when too many backends fail, forcing traffic away from a degraded site.

Example: two data centres each run their own Nexus node. Site A's pool covers 198.51.100.0/24. When Site A's backends drop below 50% healthy, Nexus withdraws the prefix and all traffic routes to Site B.

gslbd.yaml on Site A:

bgp:
  enabled: true
  localASN: 65100
  routerID: "10.1.0.1"
  listenPort: 179
  holdTime: "90s"
  keepAliveTime: "30s"
  peers:
    - remoteASN: 65000
      remoteAddr: "10.1.0.254"
  poolPrefixes:
    - poolID: "site-a-web"         # ID from GET /api/v1/pools
      prefixes:
        - "198.51.100.0/24"
      withdrawThreshold: 0.5       # withdraw if <50% healthy

Finding pool IDs:

Pool IDs are assigned when pools are created via the API. Retrieve them:

# via REST API
curl -s -H "Authorization: Bearer $TOKEN" http://localhost:8080/api/v1/pools \
  | jq '.[] | {id, name}'

# via gslbctl TUI — navigate to Pools tab, the ID column shows the UUID
gslbctl

Multiple pools, multiple prefixes:

bgp:
  poolPrefixes:
    - poolID: "api-prod"
      prefixes:
        - "203.0.113.128/25"
      withdrawThreshold: 0.33    # withdraw if fewer than 1-in-3 healthy
    - poolID: "web-prod"
      prefixes:
        - "203.0.113.0/25"
      withdrawThreshold: 0.5
    - poolID: "cdn-origin"
      prefixes:
        - "198.51.100.0/24"
        - "198.51.101.0/24"      # multiple prefixes per pool
      withdrawThreshold: 0        # any healthy endpoint = keep routes up

Scenario 4 — Combined: node VIP + pool prefixes

The most complete setup: the node VIP keeps DNS reachable via anycast, and pool prefixes inject/withdraw based on backend health.

bgp:
  enabled: true
  localASN: 65001
  routerID: "10.0.0.1"
  listenPort: 179
  holdTime: "90s"
  keepAliveTime: "30s"
  peers:
    - remoteASN: 65000
      remoteAddr: "10.0.0.254"
  nodePrefixes:
    - "203.0.113.0/24"         # DNS anycast VIP — always up
  poolPrefixes:
    - poolID: "web-prod"
      prefixes:
        - "198.51.100.0/24"    # app anycast VIP — health-driven
      withdrawThreshold: 0.5

Peer configuration examples

FRR (Linux)

Install FRR and enable BGP:

apt install frr
# edit /etc/frr/daemons, set bgpd=yes
systemctl restart frr

/etc/frr/frr.conf:

frr defaults traditional
hostname router
log syslog informational
!
router bgp 65000
 bgp router-id 10.0.0.254
 no bgp ebgp-requires-policy
 !
 neighbor 10.0.0.1 remote-as 65001
 neighbor 10.0.0.1 description nexus-lon-01
 !
 address-family ipv4 unicast
  neighbor 10.0.0.1 activate
  neighbor 10.0.0.1 soft-reconfiguration inbound
 exit-address-family
!

Verify:

vtysh -c "show bgp summary"
vtysh -c "show bgp ipv4 unicast"

Expected output when Nexus is up:

Neighbor        V    AS MsgRcvd MsgSent   Up/Down  State/PfxRcd
10.0.0.1        4 65001       8       6 00:02:14         1

FRR with MD5 authentication

Add the password field to the Nexus peer config and a matching password statement in FRR:

gslbd.yaml:

bgp:
  peers:
    - remoteASN: 65000
      remoteAddr: "10.0.0.254"
      password: "s3cr3t-bgp-pw"

/etc/frr/frr.conf:

 neighbor 10.0.0.1 remote-as 65001
 neighbor 10.0.0.1 password s3cr3t-bgp-pw

BIRD 2

/etc/bird/bird.conf:

router id 10.0.0.254;

protocol kernel {
  ipv4 { export all; };
}

protocol device {}

protocol bgp nexus_lon01 {
  local 10.0.0.254 as 65000;
  neighbor 10.0.0.1 as 65001;
  description "Nexus GSLB lon-01";
  ipv4 {
    import all;
    export none;
  };
}

Verify:

birdc show protocols nexus_lon01
birdc show route

Cisco IOS

router bgp 65000
 neighbor 10.0.0.1 remote-as 65001
 neighbor 10.0.0.1 description nexus-gslb
 !
 address-family ipv4 unicast
  neighbor 10.0.0.1 activate
  no auto-summary
  no synchronization
 exit-address-family

Verify:

show bgp summary
show bgp ipv4 unicast neighbors 10.0.0.1 received-routes

VyOS

set protocols bgp system-as 65000
set protocols bgp neighbor 10.0.0.1 remote-as 65001
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast

`withdrawThreshold` decision table

Threshold	Withdraw when...	Use case
`0.0` (default)	All endpoints are down	Maximum tolerance — keep routes up as long as a single backend responds
`0.25`	Fewer than 25% healthy	Withdraw early to avoid an overwhelmed survivor
`0.5`	Fewer than 50% healthy	Standard 50% threshold — balanced
`1.0`	Any endpoint is unhealthy	Strict — all-or-nothing; withdraw if a single endpoint fails

Example — avoid sending traffic to a nearly-dead site:

poolPrefixes:
  - poolID: "web-prod"
    prefixes: ["198.51.100.0/24"]
    withdrawThreshold: 0.4   # withdraw if fewer than 40% of backends respond

Graceful shutdown behaviour

When the Nexus daemon receives SIGINT or SIGTERM, the shutdown sequence is:

DNS server stops accepting new queries.
BGP calls StopBgp with a 10-second timeout.
GoBGP sends a BGP NOTIFICATION (reason: cease) to each peer.
The peer removes all routes learned from this session from its RIB.
The daemon exits.

From the upstream router's perspective, routes disappear within one BGP convergence cycle — typically well under a second for directly connected peers with a short hold timer.

!!! tip "Tune hold timer for faster failover" Reducing holdTime to "9s" and keepAliveTime to "3s" means an unresponsive Nexus node is detected within 9 seconds (the minimum allowed by RFC 4271). This is the minimum reasonable value for reliable convergence.

```yaml
bgp:
  holdTime: "9s"
  keepAliveTime: "3s"
```

For graceful planned maintenance, the SIGTERM-based shutdown is always faster than hold-timer expiry.

Troubleshooting

Session stuck in Active state

The Active state means GoBGP is attempting to connect but the peer is not responding.

Check that port 179 is open on this node: ss -tlnp | grep 179
Check that the peer's firewall allows TCP/179 inbound from this node's IP.
Confirm remoteAddr matches the peer's actual listening interface.
If using MD5, ensure the password matches exactly on both sides — a mismatch causes the TCP handshake to fail silently.

Session establishes but routes are not received by the peer

Confirm localASN matches what the peer expects (remote-as on the peer config).
On FRR, check no bgp ebgp-requires-policy is set, or add explicit import/export policies.
Run vtysh -c "show bgp ipv4 unicast neighbors 10.0.0.1 advertised-routes" to see what GoBGP is sending.

Pool prefix never announced

Verify the poolID matches exactly (it is a UUID, not the pool's name). Use GET /api/v1/pools to check.
Confirm the pool has health checks configured and at least one member is enabled.
Check daemon logs for bgp: pool prefix announced — if absent, either no health data has been received yet, or the ratio is below withdrawThreshold.
Set logging.level: debug to see each health sink call.

Node prefix announced but not in the upstream RIB

Check show bgp ipv4 unicast on the peer: is the prefix in the BGP table but not in the routing table? The peer may have a route policy rejecting it. Add no bgp ebgp-requires-policy (FRR) or an explicit network statement.
Confirm routerID is a valid IPv4 address reachable by the peer.

bgp.localASN must be non-zero error at startup

The localASN field was not set or was set to zero. It is required when bgp.enabled: true.

Binary is large after adding BGP

GoBGP embeds gRPC and protobuf, which adds approximately 16 MB to the stripped binary. This is expected. The binary remains CGO-free and cross-compiles for linux/amd64 with CGO_ENABLED=0.

Security considerations

MD5 TCP authentication

BGP MD5 (RFC 2385) prevents session hijacking by signing TCP segments. It is not encryption — use it on untrusted networks, not as a substitute for proper network segmentation.

bgp:
  peers:
    - remoteASN: 65000
      remoteAddr: "10.0.0.254"
      password: "long-random-string-here"

AS path filtering on the peer

Your upstream router should filter BGP announcements to only accept prefixes you own. Example FRR prefix list:

ip prefix-list NEXUS-ALLOWED seq 10 permit 203.0.113.0/24
ip prefix-list NEXUS-ALLOWED seq 20 permit 198.51.100.0/24
ip prefix-list NEXUS-ALLOWED seq 99 deny 0.0.0.0/0 le 32

router bgp 65000
 address-family ipv4 unicast
  neighbor 10.0.0.1 prefix-list NEXUS-ALLOWED in

This ensures a misconfigured Nexus cannot accidentally announce prefixes it has no authority over.

Private ASN in production

If your deployment is entirely within a private network (RFC 1918 addresses, no upstream internet connectivity), private-use ASNs (64512–65534) are appropriate and do not require IANA allocation.

For deployments that peer with internet-facing routers, use a registered ASN from your Regional Internet Registry (ARIN, RIPE, APNIC, etc.).

BGP Route Health Injection

BGP Route Health Injection Setup

Concepts

Scenario 1 — Single node, anycast VIP for the GSLB service

Scenario 2 — Three-node anycast cluster (ECMP)

Scenario 3 — Pool-driven prefix injection

Scenario 4 — Combined: node VIP + pool prefixes

Peer configuration examples

FRR (Linux)

FRR with MD5 authentication

BIRD 2

Cisco IOS

VyOS

withdrawThreshold decision table

Graceful shutdown behaviour

Troubleshooting

Security considerations

`withdrawThreshold` decision table