Google recently published an incident report for a GCP zonal outage that happened back on October 19. This involved a power outage at a datacenter, which sounds like it’s less directly relevant for those of us who use clouds rather than run them. But the details of the incident reveal how it’s a great example of the nature of complex systems failure.
There’s a surprising amount of detail packed into the relatively short write-up. I’ll recapitulate the bits I find most interesting here. As always, though, I recommend you read the original first.
Power distribution in europe-west-2b
Here’s my attempt at a high-level representation of how power is delivered to the GCP eureope-west-2b datacenter involved in the incident.
This is very hand-wavy because:
- I don’t have any domain expertise in datacenter power distribution
- The only info I have on the datacenter power topology is what’s in the original write-up
- I’m not showing any detail about how the switchboard feeds multiple circuits (for the two reasons stated above, and because it isn’t germane to the failure here)
The part that I have least confidence in is the block that I’ve labeled “UPS subsystem (battery backup)”. I’m using it to
- indicates that there’s a battery-backup UPS that feeds the data center load if power is interrupted, and
- there are multiple power sources that can potentially feed the load, which somehow interconnect
My goal with this diagram is to give the reader a visual sense of the following facts:
- The datacenter had two independent power feeds that come from the primary utility provider, which are labelled “feed A” and “feed B”. They refer to this as a 2N architecture.
- Each of these two feeds terminates into a separate power subsystem called a switchboard
- If one of the feeds fails, the switchboards can be bridged (indicated by the open switch between them), which enables the datacenter to be powered by a single feed
- There are two (redundant) programmable logic controllers (PLCs) which can automatically activate emergency diesel generators. The PLCs will activate the generators if they sense that both feeds are no longer delivering power. Each PLC reads from two voltage sensors, one on each feed.
- There is an uninterruptible power supply (UPS) system – a set of battery banks that provide a few minutes of power to give the emergency generators time to come online in the event that the two feeds both fail.
- The datacenter is also served by a standby utility provider. Manual intervention is required to switch to using the standby power
If you aren’t familiar with PLCs, they are basically simple embedded processors that are used in industrial control systems.
Now, let’s get into what happened here.
A load test reveals an issue with feed B, a breaker is opened to fix it
Six days before impact
On October 14, 2026, Google engineers run their quarterly generator load test. The test reveals an issue that requires corrective maintenance (the report doesn’t say what this issue is).
The engineers open the circuit breaker on feed B in order to perform the maintenance, and configure the switchboards in the mode where a single feed can power the datacenter:
Everything is still working fine at this point. However, one level of redundancy has been temporarily eliminated so the maintenance can happen.
An unrelated cable fault takes out feed A
Seven minutes before impact
On October 19, 2206, at 6:53 BST, something goes wrong with an underground cable on feed A, which interrupts the flow of power on that feed.
Sensors show good voltage, the generators stay off
With the fault on feed A and the breaker open on feed B, the datacenter is not receiving power from either of the two feeds. With the power interruption, the UPS system kicks in, delivering several minutes of power in anticipation of the emergency generators coming online.
The expected behavior in this scenario is for the PLCs to detect that power has been interrupted on both feeds and to start those generators. The PLCs are configured to trigger the generators if they detect that the voltages on both feeds are too low. However, while voltage sensor A detects a drop in voltage, voltage sensor B does not.
The issue is not with the sensor: it correctly indicates that the line on feed B is still energized. Rather, the issue is in the configuration of the PLC itself. It is configured to trigger the generators on undervoltage of the two lines. But, as this case demonstrates, expected voltage on the feed does not guarantee that power is being delivered. (After the incident, Google engineers will change the configuration so that the PLC also takes into account the state of the breaker).
Even though there are two PLCs, they are configured to use identical logic, and hence both of them conclude that feed B is still delivering power.
As a result, the diesel generators never activate.
The UPS is exhausted
Impact starts
At 07:00 BST, the batteries in the UPS start to become depleted, and parts of the data center lose power. As a result, the services that were previously running in this zone no longer respond to requests. There is now an incident.
The response begins: the stubbornness of a PLC
Four minutes after impact starts
The failures trigger alerts, and the Google engineers spring into action.
Because the PLC supports receiving commands via remote connections, engineers at the datacenter connect to the PLC and attempt to transfer the load to the diesel generator. But because the PLC sees expected voltage levels on sensor B, it does not allow this transfer to occur: the engineers can’t use it to switch power to the emergency generators.
Workloads are automatically rerouted to other zones
Nine minutes after impact
With services in the europe-west2-b zone no longer being reachable, GCP’s systems automatically fail over the impacted traffic and workloads to other zones in the europe-west2 region. As a result, these other zones get slammed with new load, and they show elevated 500 and 503 API error rates as the compute resources in these zones become saturated.
The responders also take action to mitigate the impact by making adjustments to regional service routing and preventing the other zones in europe-west2 from making calls into eureope-west2-b.
The standby utility provider comes online
Fifty-four minutes after impact starts
With the failure to bring the emergency generators online, the engineers turn to the standby utility provider as a source of power. This strategy succeeds, and at 7:54 BST, power is restored.
Multiple contributing factors
One lesson I always try to drive home when discussing incidents is the fact that complex systems failures require multiple conditions to be present at the same time. Here are the conditions mentioned in the writeup:
- The underground cable fault that happened on feed A
- Configuration of the PLC that led to it not triggering when the breaker was open
- The problem on feed B that warranted corrective maintenance
- Running the quarterly load test that discovered the problem (!)
- The breaker opened on feed B in order to perform the maintenance
- The PLC not supporting shifting load if it detected one of the feeds being active
In particular, note how the datacenter had four separate sources of power:
- Feed A
- Feed B
- Emergency diesel generators
- Standby utility provider
Here they experienced three(!) co-incident failures:
- The underground cable fault
- The problem detected by the quarterly load test
- The misconfigured PLC
Fighting against the automation
In addition to the failures above, there was the issue of not being able to manually trigger a failover to the emergency generators. I cannot imagine the frustration that Google engineers must have felt at the PLC not permitting them to redirect the load to the emergency generators(!).
Yet again, saturation rears its ugly head again
Here we see two forms of saturation at play during the incdient:
- The datacenter running out of power (an essential resource)
- The adjacent zones getting saturated as traffic was rerouted to them
Another case for my conjecture
Finally, this is another data point in support of my conjecture on why reliable systems fail:
Once a system reaches a certain level of reliability, most major incidents will involve:
- A manual intervention that was intended to mitigate a minor incident, or
- Unexpected behavior of a subsystem whose primary purpose was to improve reliability
This is a clear example of the first scenario, where the manual intervention was opening the breaker on feed B in order to perform maintenance. (Although, technically, it doesn’t sound like the manual intervention was intended to mitigate a minor incident, but I’m still going to count it). In this case, that eliminated a source of redundancy that left the datacenter vulnerable to a confluence of other events.
I think you can make an argument for the second one as well, when the PLC could not be used to shift traffic to the emergency generators, because the sensor indicated expected voltage on feed B. Fortunately, the engineers had another source of power available to them.
I suspect that, if they had not had access to the standby utility, they would have figured out a way to shift the load to the emergency diesel generators, but it would have taken them much longer.
