DNS Architecture for Multi-Region Sites: Anycast, GeoDNS and Failover

We spent three hours debugging a traffic imbalance where 80% of our European users were landing on our US-East origin server, despite running identical, low-latency deployments in Frankfurt and Dublin. The culprit was neither a BGP route flap nor a misconfigured Nginx upstream—it was a major European ISP's recursive resolver ignoring EDNS Client Subnet (ECS) data and caching a 30-second TTL record for nearly an hour.

DNS is an eventually consistent, distributed cache managed by third parties who routinely ignore RFC specifications to save bandwidth. If your multi-region disaster recovery or traffic steering plan relies solely on low TTLs and standard A/AAAA record switches, you are operating on borrowed time.

The False Promise of Fast DNS Failover

When an entire cloud region drops off the internet, your immediate operational goal is to stop sending traffic to those IP addresses. The textbook answer is lowering record Time-To-Live (TTL) values to 10 or 30 seconds and running an automated script to swap A records when health checks fail.

In production, this strategy breaks down immediately due to resolver behavior outside your control. Many consumer ISPs hardcode minimum TTL floors inside their recursive resolvers (often 300 to 3600 seconds) to reduce outbound DNS traffic. Furthermore, enterprise proxy networks and corporate DNS servers frequently strip EDNS options entirely.

You can verify how recursive resolvers handle your record TTLs and subnet options using dig:

class="text-code-comment"># Querying a public recursive resolver with explicit EDNS Client Subnet options
dig @8.8.8.8 app.pajdigitallabs.com +subnet=195.154.0.0/24 +ttlunits
class="text-code-comment">
# Response snippet showing remaining cache lifetime and ECS scope
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
; CLIENT-SUBNET: 195.154.0.0/24/24
;; QUESTION SECTION:
;app.pajdigitallabs.com.    IN  A

;; ANSWER SECTION:
app.pajdigitallabs.com. 12S IN  A   151.101.1.69

If a region dies while an ISP resolver has cached your record with a forced 15-minute floor, that quarter of your user base will receive hard connection timeouts until the cache expires. DNS is an announcement mechanism, not an execution mechanism.

Anycast at the Edge: Routing at Layer 4

Anycast addresses the fundamental limitation of DNS failover by decoupling traffic routing from DNS caching. Instead of changing the IP address returned to the client, Anycast uses the same IP address (or prefix) announced from multiple geographical locations simultaneously using BGP (Border Gateway Protocol).

When a client queries an Anycast IP, the internet's core routers send packets along the shortest BGP AS-Path. If a network path breaks, intermediate routers converge on the next best path in seconds, without client-side DNS involvement.

We run bird on our edge routers to manage BGP sessions directly with upstream transit providers. Below is a trimmed configuration that announces a /24 IPv4 block only when local health checks pass:

protocol device {
    scan time 10;
}

protocol static anycast_routes {
    ipv4 { table master4; };
class="text-code-comment">    # Only announce our public prefix if the dummy interface is up
    route 198.51.100.0/24 via class="text-code-string">"lo";
}

protocol bgp upstream_provider_a {
    local as 65500;
    neighbor 192.0.2.1 as 64512;
    
    ipv4 {
        import none;
        export where proto = class="text-code-string">"anycast_routes";
    };
class="text-code-comment">    
    # Enable BFD for sub-second link failure detection
    bfd graceful;
}

Anycast is exceptionally fast at network-level failover, but it is fundamentally blind to Layer 7 health. A BGP router will happily forward TCP SYN packets to an IP address even if the application server listening on port 443 is dead or returning HTTP 500 errors.

If you do not withdraw the BGP prefix locally when application health fails, Anycast will route traffic directly into a black hole.

GeoDNS and the EDNS Client Subnet (ECS) Trap

Where Anycast routes packets based on network topology, GeoDNS routes traffic based on geography. GeoDNS authoritative servers inspect the IP address making the request and return location-tailored A/AAAA records (e.g., returning 198.51.100.10 for European requests and 203.0.113.10 for Asian requests).

The fatal flaw of traditional GeoDNS is that the authoritative server sees the IP address of the recursive resolver, not the end client. If an end user in Tokyo uses a corporate recursive resolver located in London, the GeoDNS server will serve the London IP, forcing the client's traffic to cross the globe twice.

RFC 7871 (EDNS Client Subnet) solves this by attaching the client's network prefix (e.g., /24 for IPv4) to the DNS query forwarded to the authoritative server.

When configuring authoritative servers like PowerDNS or dnsdist, you must explicitly account for ECS parsing and fallback rules when ECS is stripped by privacy-focused resolvers:

-- dnsdist.conf: Route incoming queries based on EDNS Client Subnet or fallback IP
newServer({address=class="text-code-string">"10.0.1.10:53", name=class="text-code-string">"eu-backend"})
newServer({address=class="text-code-string">"10.0.2.10:53", name=class="text-code-string">"us-backend"})

addAction(AllRule(), LuaAction(function(dq)
    local ecs = dq:getEDNSClientSubnet()
    local client_ip = dq.remoteAddr:toString()
    
    if ecs then
        client_ip = ecs:getServerKey():toString()
    end
    
    -- Perform GeoIP lookup on client_ip instead of recursive resolver IP
    local country = ffi.string(geoip_lookup(client_ip))
    
    if country == class="text-code-string">"DE" or country == class="text-code-string">"FR" or country == class="text-code-string">"GB" then
        return Act.Pool, class="text-code-string">"eu-backend"
    else
        return Act.Pool, class="text-code-string">"us-backend"
    end
end))

Relying purely on GeoDNS assumes that ECS coverage is 100%. In reality, ECS adoption hovers around 70-85% globally depending on the region. Privacy tools, Apple Private Relay, and standard VPNs strip ECS entirely, making GeoDNS a heuristic rather than a deterministic router.

Comparing Strategy Architectures

Choosing the wrong DNS strategy introduces subtle operational traps. The table below outlines how these approaches compare under production failure modes:

| Metric / Feature | Standalone GeoDNS | Pure BGP Anycast | Latency-Based Unicast | Hybrid (Anycast Edge + L7 Dynamic Upstream) | | :--- | :--- | :--- | :--- | :--- | | P99 Failover Speed | 5 mins – 1 hour | 1 – 3 seconds | 5 mins – 30 mins | < 1 second | | App-Layer Failure Awareness | No (Requires API updates) | No (Requires BGP withdraw) | No (Requires API updates) | Yes (Edge proxies re-route) | | ECS / ISP Dependency | High | None | High | None (Terminated at Edge) | | Setup Complexity | Low | Very High (ASN, BGP) | Medium | High | | Cost | Low | High (Hardware/Transit) | Medium | High | | Blast Radius of Bad Route | Regional | Global | Regional | Controlled via Edge Rules |

Implementing Hysteresis in Automated Failover

The most dangerous failure mode in multi-region infrastructure is "flapping"—when Region A experiences a load spike, causing its health check to fail. The automated DNS automation rapidly updates records to shift 100% of traffic to Region B. Region B, instantly overwhelmed by double its capacity, crashes. The health system detects Region B is dead and shifts traffic back to Region A, creating a catastrophic loop.

To prevent cascading failures, failover daemons must enforce hysteresis: stateful tracking, error budgets, dampening timers, and threshold limits.

Below is an operational Python script illustrating stateful dampening for dynamic record management via API:

import time
import requests

class FailoverController:
    def __init__(self, primary_ip, secondary_ip, failure_threshold=3, recovery_threshold=5):
        self.primary_ip = primary_ip
        self.secondary_ip = secondary_ip
        self.failure_threshold = failure_threshold
        self.recovery_threshold = recovery_threshold
        
        self.consecutive_failures = 0
        self.consecutive_successes = 0
        self.active_ip = primary_ip

    def check_health(self) -> bool:
        try:
            r = requests.get(fclass="text-code-string">"http://{self.primary_ip}/healthz", timeout=2.0)
            return r.status_code == 200
        except requests.RequestException:
            return False

    def evaluate_state(self):
        is_healthy = self.check_health()

        if not is_healthy:
            self.consecutive_failures += 1
            self.consecutive_successes = 0
            print(fclass="text-code-string">"Health check failed ({self.consecutive_failures}/{self.failure_threshold})")
        else:
            self.consecutive_successes += 1
            self.consecutive_failures = 0
class="text-code-comment">
        # State transition: Healthy -> Unhealthy
        if self.active_ip == self.primary_ip and self.consecutive_failures >= self.failure_threshold:
            print(fclass="text-code-string">"CRITICAL: Switching active target to {self.secondary_ip}")
            self.update_dns(self.secondary_ip)
            self.active_ip = self.secondary_ip
class="text-code-comment">
        # State transition: Unhealthy -> Healthy (Requires longer stability window)
        elif self.active_ip == self.secondary_ip and self.consecutive_successes >= self.recovery_threshold:
            print(fclass="text-code-string">"RECOVERED: Switching active target back to {self.primary_ip}")
            self.update_dns(self.primary_ip)
            self.active_ip = self.primary_ip

    def update_dns(self, new_target_ip):
class="text-code-comment">        # Implementation interacts with DNS Provider API (Cloudflare, Route53, NS1)
class="text-code-comment">        # Enforces dampening locks to prevent API rate limiting and flapping
        pass

if __name__ == class="text-code-string">"__main__":
    controller = FailoverController(primary_ip=class="text-code-string">"198.51.100.50", secondary_ip=class="text-code-string">"203.0.113.50")
    while True:
        controller.evaluate_state()
        time.sleep(5)

The Production Blueprint We Use

We stopped treating DNS as a dynamic load balancer. Instead, we split traffic steering into two distinct architectural tiers:

1. Layer 4 / Layer 7 Anycast Edge: We operate (or leverage) an Anycast edge layer. The public A/AAAA records point to Anycast IP addresses that almost never change. DNS caching issues become irrelevant because the IP address resolving to the client is static. 2. Dynamic Upstream Routing: The Anycast edge nodes terminate TLS connections and inspect HTTP requests directly. These edge nodes maintain persistent HTTP/2 or HTTP/3 pools to our actual backend origins in eu-central-1, us-east-1, and ap-southeast-1.

If eu-central-1 fails, the Anycast edge node detects the upstream TCP reset or 5xx surge instantly. It shifts origin traffic to us-east-1 over private backbone links within milliseconds. The client never drops the TLS session, and no DNS query is executed.

GeoDNS is reserved strictly for initial bootstrap routing—directing the Anycast edge proxies to the nearest origin cluster—not for high-availability failover.

Engineering Judgement

If you need sub-minute failover across regions, do not rely on changing DNS records. The caching behavior of intermediate recursive resolvers makes DNS dynamic failover inherently non-deterministic.

Use Anycast for ingress stability, terminate TLS at the edge, and push application health checks and failover decisions down into L7 reverse proxies or service meshes operating behind static IP addresses. Keep your DNS static, keep your TTLs reasonable, and move path management into the network layer where you control the data plane.