Skip to content

Graceful Restart

Jump to bottom
Thomas Mangin edited this page Nov 13, 2025 · 4 revisions

Graceful Restart

BGP Graceful Restart (GR) enables BGP sessions to maintain forwarding state during control plane restarts, minimizing service disruption. ExaBGP provides full RFC 4724 implementation, supporting both Helper and Restarter roles for high-availability BGP deployments.

Table of Contents

What is Graceful Restart?

Graceful Restart (GR) is a BGP capability defined in RFC 4724 that allows a BGP speaker to preserve forwarding state while the BGP control plane restarts. This minimizes traffic loss during planned maintenance, software upgrades, or control plane failures.

The Problem Without Graceful Restart

When a BGP session goes down, the traditional behavior is:

  1. BGP neighbor detects session failure (keepalive timeout)
  2. All routes from that neighbor are immediately withdrawn
  3. Forwarding stops → Traffic blackhole
  4. Session re-establishes
  5. All routes are re-learned
  6. Forwarding resumes

Result: Service disruption of 30-180 seconds depending on convergence time.

The Solution With Graceful Restart

With Graceful Restart enabled:

  1. BGP session goes down (control plane restart)
  2. Forwarding state is preserved (routes remain in RIB/FIB)
  3. Helper peer marks routes as "stale" but keeps forwarding
  4. Session re-establishes within restart time
  5. Routes are refreshed
  6. Stale routes are removed

Result: Minimal to zero traffic loss (typically <5 seconds).

How Graceful Restart Works

Graceful Restart Handshake

[Session Establishment]

ExaBGP (Restarter)                     BGP Peer (Helper)
       │                                       │
       │──── OPEN with GR Capability ─────────▶│
       │     (Restart Time: 120s)               │
       │     (Families: ipv4/unicast)           │
       │                                       │
       │◀──── OPEN with GR Capability ─────────│
       │     (Restart Time: 120s)               │
       │                                       │
       │──── KEEPALIVE ───────────────────────▶│
       │◀──── KEEPALIVE ───────────────────────│
       │                                       │
       │  BGP session established              │
       │  Routes exchanged normally            │

Graceful Restart Sequence

[Control Plane Restart Scenario]

ExaBGP (Restarter)                     BGP Peer (Helper)
       │                                       │
       │  ExaBGP process restarts              │
       │  (planned upgrade, crash)             │
       │                                       │
       ├─ TCP session closes ───────────────▶ │
       │                                       │
       │                              Helper detects
       │                              session loss
       │                              └─ Starts GR timer
       │                                 (120s restart time)
       │                              └─ Marks routes STALE
       │                              └─ KEEPS forwarding!
       │                                       │
       │  ExaBGP restarts                      │
       │                                       │
       │──── TCP SYN ─────────────────────────▶│
       │◀──── TCP SYN-ACK ─────────────────────│
       │──── OPEN (with GR, R-bit set) ───────▶│
       │     (R-bit = "I'm restarting")        │
       │                                       │
       │◀──── OPEN with GR ────────────────────│
       │                                       │
       │──── KEEPALIVE ───────────────────────▶│
       │◀──── KEEPALIVE ───────────────────────│
       │                                       │
       │◀──── Route Refresh Request ───────────│
       │      (refresh stale routes)           │
       │                                       │
       │──── UPDATE (all routes) ──────────────▶│
       │──── END-OF-RIB marker ────────────────▶│
       │                                       │
       │                              Helper removes
       │                              stale routes
       │                              └─ Forwarding updated
       │                                       │
       │  Graceful Restart complete            │

Key Points:

  • Helper peer preserves forwarding for restart-time seconds (configurable)
  • Restarter sets R-bit in OPEN message after restart
  • Helper requests route refresh to update stale routes
  • Stale routes removed after END-OF-RIB marker received

ExaBGP GR Capabilities

ExaBGP provides full RFC 4724 Graceful Restart implementation:

✅ Both GR Roles:

  • Restarter: ExaBGP advertises GR capability and can perform graceful restart
  • Helper: ExaBGP supports peers that perform graceful restart

✅ Per-Address-Family Support:

  • IPv4 unicast
  • IPv6 unicast
  • VPNv4 / VPNv6
  • FlowSpec
  • EVPN
  • All supported address families

✅ GR Capability Negotiation:

  • Restart Time advertisement (seconds)
  • Per-AFI/SAFI forwarding state flags
  • R-bit (restart state) flag handling

✅ Stale Route Handling:

  • Marks routes as stale during GR period
  • Removes stale routes after convergence
  • END-OF-RIB marker support (RFC 4724)

Implementation: src/exabgp/bgp/message/open/capability/graceful.py

RFC Compliance: RFC 4724 fully implemented

Graceful Restart Modes

1. Restarter Mode

ExaBGP is the speaker performing the restart.

When ExaBGP restarts (planned or unplanned):

  1. BGP session closes
  2. Peer (Helper) preserves routes for restart-time
  3. ExaBGP re-establishes session with R-bit set
  4. ExaBGP re-announces all routes
  5. Peer removes stale routes

Configuration:

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 120;  # Restart time: 120 seconds
    }

    family {
        ipv4 unicast;
    }
}

When to Use:

  • ExaBGP process upgrades
  • ExaBGP host maintenance
  • Control plane restarts (while forwarding continues elsewhere)

2. Helper Mode

ExaBGP's peer is performing the restart.

When the peer restarts:

  1. ExaBGP detects session loss
  2. ExaBGP marks peer's routes as stale
  3. ExaBGP keeps routes in its RIB (does NOT manipulate FIB - external process decides)
  4. Peer re-establishes with R-bit
  5. ExaBGP sends Route Refresh request
  6. Peer re-announces routes
  7. ExaBGP removes stale routes

Configuration:

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 120;
    }

    family {
        ipv4 unicast;
    }
}

When to Use:

  • Peer router upgrades (Cisco, Juniper, etc.)
  • Peer restarts while forwarding plane continues
  • High-availability BGP deployments

Important: ExaBGP does NOT manipulate the FIB (Forwarding Information Base). External API processes receive route updates and must decide whether to maintain forwarding during graceful restart.

Configuration

Basic Graceful Restart

Enable GR with 120-second restart time:

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 120;  # Restart time in seconds
    }

    family {
        ipv4 unicast;
        ipv6 unicast;
    }
}

Graceful Restart with Multiple Address Families

GR applies to all configured address families:

neighbor 2001:db8::1 {
    router-id 192.168.1.2;
    local-address 2001:db8::2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 180;  # 3-minute restart window
    }

    family {
        ipv4 unicast;
        ipv6 unicast;
        ipv4 flow;     # FlowSpec also GR-capable
    }
}

Disable Graceful Restart

Disable GR (default if not configured):

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    # No graceful-restart directive = GR disabled

    family {
        ipv4 unicast;
    }
}

Configuration Parameters

Parameter Type Default Description
graceful-restart Integer (seconds) Disabled Restart time (0-4095 seconds)

Recommended Values:

  • Small networks: 60-120 seconds
  • Medium networks: 120-180 seconds
  • Large networks: 180-300 seconds

Considerations:

  • Longer restart time = more resilience to slow restarts
  • Longer restart time = stale routes persist longer if restart fails
  • Typical: 120 seconds (2 minutes)

Use Cases

1. ExaBGP Software Upgrades (Zero Downtime)

Scenario: Upgrade ExaBGP to a new version without service disruption.

Architecture:

ExaBGP (Anycast DNS)           BGP Router (Helper)
       │                              │
       │──── Announces 8.8.8.8 ───────▶│
       │                              │
       │  (upgrade ExaBGP)             │
       │  session closes               │
       │                              │
       │                       Router keeps
       │                       forwarding to
       │                       8.8.8.8 (GR)
       │                              │
       │  (ExaBGP restarts)            │
       │◀──── Restore session ─────────│
       │──── Re-announce 8.8.8.8 ──────▶│
       │                              │
       │  Zero packet loss!            │

Configuration:

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 120;  # 2-minute upgrade window
    }

    family {
        ipv4 unicast;
    }
}

Benefits:

  • Zero service disruption during upgrades
  • No need for redundant ExaBGP instances
  • Simple upgrade workflow

2. High-Availability Anycast Services

Scenario: Anycast service with graceful failover.

Architecture:

           ┌─────────────┐
           │ BGP Router  │
           │  (Helper)   │
           └─────────────┘
                  │
         ┌────────┴────────┐
         │                 │
    ┌─────────┐       ┌─────────┐
    │ ExaBGP  │       │ ExaBGP  │
    │ Node 1  │       │ Node 2  │
    │ 1.1.1.1 │       │ 1.1.1.1 │
    └─────────┘       └─────────┘
         │                 │
    [Web Server]      [Web Server]

Both nodes advertise the same anycast IP with GR enabled. During restarts, the router maintains forwarding to the restarting node while its forwarding plane continues serving traffic.

3. FlowSpec DDoS Mitigation Continuity

Scenario: Maintain DDoS filtering rules during ExaBGP restart.

Architecture:

ExaBGP (FlowSpec)              Router (Helper + Enforcer)
       │                              │
       │── FlowSpec: Block attacker ──▶│
       │   (drop src 1.2.3.4)          │
       │                              │
       │                       Router applies
       │                       filter in hardware
       │                              │
       │  (ExaBGP restarts)            │
       │  session closes               │
       │                              │
       │                       Router KEEPS
       │                       filtering (GR)
       │                       └─ Attacker still
       │                          blocked!
       │                              │
       │  (ExaBGP reconnects)          │
       │── Re-send FlowSpec rules ─────▶│
       │                              │
       │  DDoS protection maintained   │

Configuration:

neighbor 192.168.1.1 {
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65000;

    capability {
        graceful-restart 120;
    }

    family {
        ipv4 flow;  # FlowSpec with GR
    }
}

Benefits:

  • DDoS filters remain active during control plane restart
  • No traffic leak window during upgrades
  • Continuous attack mitigation

4. Route Reflector Client Restart

Scenario: ExaBGP as route reflector client with graceful restart support.

Architecture:

    Route Reflector
    (Helper Mode)
           │
           │
      ┌────┴────┐
      │         │
   ExaBGP    ExaBGP
   Client 1  Client 2
   (GR)      (GR)

Configuration:

neighbor 192.168.1.254 {  # Route Reflector
    router-id 192.168.1.2;
    local-address 192.168.1.2;
    local-as 65001;
    peer-as 65001;  # iBGP

    capability {
        graceful-restart 180;
    }

    family {
        ipv4 unicast;
        ipv6 unicast;
        ipv4 vpn;
    }
}

Graceful Restart Process

State Machine

[GR States]

IDLE
  │
  └──▶ CONNECT
         │
         └──▶ OPEN_SENT (GR capability advertised)
                │
                └──▶ OPEN_CONFIRM (GR negotiated)
                       │
                       └──▶ ESTABLISHED
                              │
                              │  [Session Loss Detected]
                              ▼
                           GR_RESTART
                              │
                              │  (Helper: preserve routes)
                              │  (Restarter: attempt reconnect)
                              │
                              ├──▶ [Timeout] ──▶ DELETE_STALE_ROUTES
                              │                        │
                              │                        └──▶ IDLE
                              │
                              └──▶ [Reconnect Success]
                                      │
                                      └──▶ ESTABLISHED (Route Refresh)
                                             │
                                             └──▶ END-OF-RIB received
                                                    │
                                                    └──▶ GR_COMPLETE

END-OF-RIB Marker

RFC 4724 defines END-OF-RIB (End-of-Routing-Information-Base) marker to signal completion of route updates after graceful restart.

Format:

  • Empty UPDATE message for IPv4 unicast
  • MP_UNREACH_NLRI with no routes for other address families

ExaBGP Behavior:

  • Sends END-OF-RIB after re-announcing all routes post-restart
  • Recognizes END-OF-RIB from peers to complete GR process
  • Removes stale routes after END-OF-RIB received

Monitoring and Verification

Via API (JSON Format)

Enable API monitoring:

[exabgp.api]
encoder = json

neighbor 192.168.1.1 {
    # ... neighbor config ...

    api {
        processes [ monitor-gr ];
    }
}

process monitor-gr {
    run python3 /etc/exabgp/api/monitor.py;
    encoder json;
}

API messages during graceful restart:

{
  "type": "notification",
  "notification": "neighbor 192.168.1.1 up",
  "neighbor": {
    "address": { "local": "192.168.1.2", "peer": "192.168.1.1" },
    "asn": { "local": 65001, "peer": 65000 }
  }
}

Capability negotiation:

{
  "type": "open",
  "neighbor": {
    "address": { "peer": "192.168.1.1" },
    "capabilities": {
      "graceful-restart": {
        "enabled": true,
        "restart-time": 120
      }
    }
  }
}

Via Logging

Enable GR logging:

[exabgp.log]
level = INFO
network = true
packets = true

Log output example:

INFO     network  Peer 192.168.1.1: Graceful Restart capability negotiated (restart-time: 120s)
INFO     network  Peer 192.168.1.1: Session established with GR
WARNING  network  Peer 192.168.1.1: Session lost - starting GR timer (120s)
INFO     network  Peer 192.168.1.1: Reconnected with R-bit set (graceful restart in progress)
INFO     network  Peer 192.168.1.1: Received END-OF-RIB - GR complete
INFO     network  Peer 192.168.1.1: Removed 0 stale routes

Verification Checklist

Before restart:

  • ✅ Verify GR capability negotiated: check OPEN messages
  • ✅ Confirm restart-time value: check logs or API
  • ✅ Test BGP session stability

During restart:

  • ✅ Verify peer preserves routes (Helper mode)
  • ✅ Verify reconnection within restart-time
  • ✅ Check R-bit set in OPEN message

After restart:

  • ✅ Verify all routes re-announced
  • ✅ Confirm END-OF-RIB sent/received
  • ✅ Verify stale routes removed

Limitations and Considerations

ExaBGP Specific Limitations

  1. No FIB Manipulation

    • ExaBGP does NOT install routes into the kernel FIB
    • External API processes must handle forwarding state preservation
    • GR only affects ExaBGP's RIB and what it advertises to peers

  2. API Process Coordination

    • API processes must independently preserve forwarding state
    • ExaBGP's restart does NOT guarantee external process continuity
    • Design external processes for independent graceful restart

  3. Restart Time Window

    • ExaBGP must re-establish session within configured restart-time
    • If exceeded, peer removes all routes → traffic loss
    • Plan for worst-case restart duration

General BGP GR Limitations

  1. Not a Replacement for Redundancy

    • GR provides restart resilience, not failure protection
    • Use redundant ExaBGP instances for true high availability
    • GR complements (not replaces) multi-homing and anycast

  2. Peer Support Required

    • Both peers must support and negotiate GR capability
    • Verify peer router supports RFC 4724
    • Check vendor-specific GR limitations

  3. Address Family Support

    • GR applies per address family
    • Not all address families may support GR on all routers
    • Verify AFI/SAFI GR support with peer equipment

  4. Stale Route Risk

    • If restart fails, stale routes may blackhole traffic
    • Peer's restart-time determines how long stale routes persist
    • Monitor GR failures and have rollback procedures

Common Errors and Solutions

Error 1: Graceful Restart Not Negotiated

Symptom: GR configured but not negotiated with peer.

Logs:

WARNING  Peer 192.168.1.1: Graceful Restart capability not negotiated

Causes:

  1. Peer does not support GR (RFC 4724)
  2. Peer has GR disabled
  3. Configuration mismatch

Solutions:

# Check peer GR configuration
# Cisco:
router bgp 65000
  neighbor 192.168.1.2 activate
  neighbor 192.168.1.2 graceful-restart

# Juniper:
protocols {
    bgp {
        group peers {
            graceful-restart;
        }
    }
}

# Verify ExaBGP config
grep graceful-restart /etc/exabgp/exabgp.conf

Error 2: Session Timeout During Restart

Symptom: Session does not re-establish within restart-time.

Logs:

ERROR    Peer 192.168.1.1: Graceful restart timeout (120s exceeded)
INFO     Peer 192.168.1.1: Removing stale routes

Causes:

  1. ExaBGP restart took longer than configured time
  2. Network issues preventing reconnection
  3. restart-time value too short

Solutions:

# Increase restart-time
neighbor 192.168.1.1 {
    capability {
        graceful-restart 300;  # 5 minutes instead of 2
    }
}

# Or optimize ExaBGP startup time
# Reduce configuration size
# Use faster storage
# Optimize API processes

Error 3: Routes Not Re-announced After Restart

Symptom: Session re-establishes but routes missing.

Causes:

  1. API process not re-announcing routes
  2. Configuration lost during restart
  3. Process startup race condition

Solutions:

# API process must re-announce all routes after restart
import sys
import time

def announce_routes():
    """Re-announce all routes after ExaBGP restart"""
    routes = [
        "announce route 10.0.0.0/8 next-hop 192.168.1.2",
        "announce route 172.16.0.0/12 next-hop 192.168.1.2",
    ]

    for route in routes:
        sys.stdout.write(f"{route}\n")
        sys.stdout.flush()

# On startup, wait for session establishment
time.sleep(5)
announce_routes()

Error 4: Peer Removes Routes Immediately

Symptom: Peer does not preserve routes during graceful restart.

Causes:

  1. Peer GR helper mode disabled
  2. Peer does not support GR for address family
  3. Session terminated ungracefully

Solutions:

# Verify peer GR helper capability
# Cisco:
show bgp neighbors 192.168.1.2 | include Graceful

# Juniper:
show bgp neighbor 192.168.1.2 | match graceful

# ExaBGP logs should show GR negotiation
[exabgp.log]
level = DEBUG
network = true

# Check for GR capability in OPEN message

See Also

References

RFCs

ExaBGP Documentation

Implementation

  • Source Code: src/exabgp/bgp/message/open/capability/graceful.py
  • Capability Type: 64 (0x40)
  • ExaBGP Version: 3.x, 4.x, 5.x/main

👻 Ghost written by Claude (Anthropic AI)

🏠 Home

🚀 Getting Started

🔧 API

🛡️ Use Cases

🌐 Address Families

⚙️ Configuration

🔍 Operations

📚 Reference

🔄 Migration

🌍 Community

🔗 External

👻 Ghost written by Claude (Anthropic AI)

Clone this wiki locally