Routine maintenance as a failure vector in modern networks

Early in my consulting career, I assumed maintenance windows reduced risk. After all, the purpose of planned maintenance is to improve reliability, apply fixes and prevent future outages. That assumption changed after I participated in what should have been a routine infrastructure change.

Every pre-check passed. Device health looked normal. High-availability synchronization was complete. Monitoring showed no obvious concerns. Yet shortly after the change, users began reporting application failures.

The root cause was not a failed upgrade, hardware fault or software defect. The maintenance activity exposed a dependency elsewhere in the traffic path that nobody had considered.

Since then, I have seen similar patterns repeatedly across enterprise environments. The change itself was rarely the problem. The problem was the assumption that the change was isolated.

Planned maintenance is intended to reduce risk, but in practice, it often introduces risk into an otherwise stable network.

Many production incidents result from routine tasks such as firewall updates, DNS changes, certificate renewals, routing adjustments, load balancer failovers, WAF updates, switch upgrades or software patches, rather than dramatic failures.

The reality is that “routine” does not equate to “low risk.” It simply means the activity has been performed before, not that the current environment will respond the same way.

Modern networks have become too interconnected for maintenance to be treated as a simple device-level task. A change to one control point can expose a dependency elsewhere in the traffic path. A firewall update can affect asymmetric return traffic. A DNS change can shift users to a data center where persistence is not aligned. A load balancer failover can expose stale ARP or MAC learning issues. A certificate renewal can cause an inspection or TLS negotiation to fail in the backend. A WAF update can block application behavior that was never visible in testing.

Failures rarely stem from the maintenance activity itself, but rather from the assumption that the change is isolated.

Why routine changes still cause outages

In traditional network operations, the unit of change was often a device: upgrade a switch, modify a router, add a firewall rule, renew a certificate or reboot an appliance. That model worked better when application traffic paths were simpler, and dependencies were easier to understand.

Today, a single user transaction may cross DNS, global traffic management, WAN routing, data center switching, firewalls, load balancers, TLS inspection points, WAF policies, API gateways and backend application tiers. Each layer may make an independent decision about availability, security, routing or session handling.

This creates a risky maintenance pattern. Teams often validate only the component they changed, not the complete traffic flow before and after the change. Devices may appear healthy, configurations may load correctly and all checks may pass, yet users can still experience failures due to a changed dependency somewhere in the end-to-end path.

Google’s Site Reliability Engineering (SRE) guidance highlights that changes remain one of the most common sources of service disruption, which is why mature organizations invest heavily in change validation, rollback planning and observability. The SRE book provides extensive discussion of change management, reliability engineering and operational risk in large-scale environments.

For this reason, maintenance windows should be evaluated as both operational events and potential failure vectors.

Common failure points during maintenance

One common issue is state mismatch. Firewalls, load balancers, NAT devices and application delivery controllers often maintain connection or session state. During failover, reboot or path change, existing flows may not survive even if the standby device becomes active as designed. New connections may succeed while long-lived sessions fail. In other cases, traffic may enter through one device and return through another, causing stateful inspection to drop packets that appear invalid.

Asymmetric routing is another frequent cause. A routing change may look harmless from a Layer 3 perspective, but if the forward and return paths traverse different firewalls or inspection zones, applications can fail intermittently. The network may still be “up,” but the security policy no longer sees the full conversation.

Layer 2 behavior is also underestimated. In highly available data center designs, MAC learning, ARP cache behavior, VLAN tagging, port channels and first-hop gateway behavior can determine whether traffic moves cleanly after a failover. A device may successfully assume an active role, but upstream switches or firewalls may still forward traffic toward the old path until tables age out or are refreshed.

DNS and GSLB changes introduce a different class of risk. Teams often test name resolution, but resolution is only the first step. The more important question is where users are being sent and whether that destination is ready to handle production traffic.

DNS resilience guidance published by the Internet Society emphasizes that successful name resolution alone does not guarantee application availability, particularly when multiple infrastructure dependencies exist behind the DNS response.

If global traffic management shifts users from one data center to another, the receiving site must have aligned firewall rules, load balancer configuration, health monitors, certificates, persistence behavior, routing advertisements and backend capacity. Otherwise, DNS sends users to a site that is not actually ready.

Certificate maintenance can also break more than the browser-facing endpoint. In many environments, TLS is terminated, re-encrypted, inspected or validated across multiple hops. Renewing a certificate on the external virtual server may not address backend certificates, intermediate chains, SNI behavior, cipher compatibility or trust stores used by inspection devices. The maintenance task may be described as a certificate renewal, but the real dependency is end-to-end TLS negotiation.

Security policy maintenance creates another risk. WAFs, IPSs, DDoS protection systems, bot defense platforms and firewall policies are designed to block abnormal behavior. But during updates, tuning changes or signature refreshes, they can also block legitimate application traffic if policy enforcement is not validated against real transaction patterns.

This is especially true for APIs, where small differences in headers, methods, payload structure or authentication flows can trigger unexpected enforcement.

The test environment problem

Many teams rely on pre-checks and test environments, but these controls are often less effective than they seem.

Pre-checks confirm device reachability, interface status, route existence, pool member availability and HA health. While necessary, these checks do not ensure production traffic will survive a path change because they focus on infrastructure rather than transaction validation.

Test environments rarely mirror production. Production environments involve real user volume, client diversity, DNS caching behavior, firewall states, certificates, backend latency and complex dependencies. A failover that succeeds in a lab may behave very differently in the real world.

This does not render testing useless, but test results should not be considered proof of production safety. They provide evidence, not a guarantee.

This challenge aligns with broader operational resilience guidance from the NIST Cybersecurity Framework, which emphasizes continuous monitoring, validation and recovery planning as critical operational capabilities.

A stronger maintenance process starts with mapping the traffic path before the window. For critical applications, teams should understand the normal ingress path, egress path, firewall zones, NAT points, load balancer virtual servers, DNS or GSLB decision points, TLS termination points, persistence requirements and backend dependencies.

The next step is defining failure expectations. What happens to existing sessions if a firewall is rebooted? Should source MAC, floating IP, ARP or upstream forwarding behavior change during a load balancer failover? How long will cached clients continue to access the old site after a DNS shift? Which clients and inspection devices validate the certificate chain when a certificate is replaced?

These questions should be addressed before the maintenance window, not during an outage.

Pre-checks should include both control-plane and data-plane evidence. Control-plane checks confirm configuration, synchronization, device health, routing tables, interface status and object availability. Data-plane checks validate real traffic movement: TCP handshakes, TLS negotiation, HTTP status codes, API responses, session persistence, source NAT behavior and return-path consistency.

During the change, monitoring should focus on symptoms that expose traffic failure early. Device CPU and interface status are useful, but they are not enough. Teams should also watch connection resets, denied firewall logs, WAF violation spikes, pool member selection failures, DNS answer changes, TCP retransmissions, backend 5xx errors and synthetic transaction results.

Rollback planning must also be precise. Simply rolling back a configuration is often insufficient. If a DNS record changes, cached clients may continue using the previous answer. If a firewall state table is cleared, restoring the rule does not recover active sessions. If failover alters forwarding behavior, upstream devices may require ARP refresh, route reconvergence or manual validation.

An effective rollback plan should identify lost state, persistent caches and the evidence required to confirm recovery.

Treating maintenance as a resilience exercise

The objective is not to make maintenance overly complex or bureaucratic. The objective is to avoid underestimating its risks.

Every maintenance window is a controlled opportunity to test whether the network behaves as specified by the architecture.

If failover is part of the design, maintenance should verify failover behavior. If a secondary data center is expected to handle traffic, maintenance should demonstrate that it can process real transactions. If security policies are updated, maintenance should prove that legitimate traffic is still allowed. If certificates are renewed, maintenance should validate the complete TLS path, not just the public endpoint.

Industry outage studies published by the Uptime Institute consistently show that human error and process failures remain significant contributors to downtime. Their annual outage research continues to highlight the role of operational processes and maintenance activities in service disruptions.

Maintenance windows provide an opportunity to identify those weaknesses before they become customer-facing incidents.

This requires closer collaboration between network, security, application and operations teams. Network engineers may own routing or load-balancing changes, but application teams understand transaction flows. Security teams understand inspection and enforcement behavior. Operations teams often see user-impacting symptoms first.

Treating maintenance as a shared traffic event rather than a device event reduces blind spots.

Routine maintenance will always involve some risk. However, the greatest risk is the false confidence that the term ‘routine’ conveys.

Modern networks fail in the spaces between systems: between DNS and load balancing, between firewalls and routing, between TLS inspection and application behavior, between HA design and actual forwarding state. Maintenance exposes those spaces.

For that reason, network teams should view every maintenance window as more than a checklist. It is a live test of architecture, operational discipline and production resilience.

This article is published as part of the Foundry Expert Contributor Network.Want to join?