Google Cloud configuration update disrupts VMware Engine stretched clusters

A faulty configuration update on Google Cloud VMware Engine (GCVE) caused a multi-region disruption on Tuesday, disrupting inter-zone connectivity across three regions.

The incident, which lasted for over ten hours, began at 5:00 PM UTC on July 14 and was resolved by 04:46 AM UTC on July 15. It affected VMware Engine stretched clusters in Sydney (australia-southeast1), Melbourne (australia-southeast2), and Frankfurt (europe-west3).

Google later identified a recent network configuration update as the cause of the inter-zone network disruption and mitigated the issue by rolling back the faulty configuration to its last-known configuration.

Google traces the fault

The first status update, posted at 08:24 PM UTC on July 14, described the incident as a network connectivity issue affecting stretched clusters, while compute and storage services remained unaffected. At that time, GCVE VMs were running as expected, but the company acknowledged that customers may experience connectivity issues with the VMs.

But soon after, the preliminary investigation indicated that the issue could be stemming from an underlying network connectivity issue affecting the infrastructure that links the zones within a stretch cluster.

“This disruption is causing synchronization issues between the affected zones, and some GCVE customers using Stretched Cluster may experience inter-site communication failures to their GCVE environments within the affected zones,” Google Cloud said in a notification.

While the company was working on restoring full connectivity, Google advised moving workloads to the healthy side of the stretched cluster, where feasible, and only after consulting Google Support.

Less than two hours after the first update, Google Cloud identified underlying inter-zone communication failures and Border Gateway Protocol (BGP) session flapping between cluster zones. “Specifically, network connectivity has been lost between the affected zones and the witness appliance. Because th…