Google Engineer's Fateful Cable Pull Causes Four-Hour Outage
A routine hardware maintenance procedure at Google's us-central1-b region went horribly wrong when an engineer unplugged every fiber they could see, causing a partial outage that lasted over four hours.
The incident occurred on September 1st, between 7:41 and 11:52 PT, and affected a portion of the G-Cloud region, resulting in 'severe network degradation and resource isolation.'
Google's service health report explained that traffic flow drop rates for resources hosted in the affected area reached 100% during the peak of the incident, making virtual machines unreachable and causing elevated packet loss.
The root cause investigation revealed that the engineer had inadvertently disconnected all fiber-optic cables across all devices within 13 minutes, preventing warnings from reaching them before complete disconnection.