Update: Jan. 12 power outage

January 17, 2022  //  FOUND IN: Updates & Resources

On Jan. 12, 2022 at approximately 8:30 a.m., facilities was performing routine scheduled maintenance on the electrical power at the North Campus Administrative Services Building main data center. 

During the transition of electrical power from one redundant power source to the other, the main circuit breaker tripped open, immediately cutting off all power including emergency back up to the data center machine floor. The circuit breaker has been replaced and a new unit has been fully tested.

This incident did not involve the backup power feeds or backup generators. These redundant systems exist in case we lose power from DTE. Instead, these systems feed power to the effected circuit breaker where the problem occurred and led to loss of power to our data systems. Routine maintenance to the Uninterrupted Power Supply Systems will continue in the future.

The organization recognizes the significant disruption any loss of power, IT and other systems cause to its mission. Preventing events like this is also a critical priority within our organization. 

Preparations for events like this limited the downtime caused by this major incident. Technology recovered within the four hours it is designed to do so, and the operational recovery was completed soon after.

Regardless of the cause, the organization still must work to prevent these events, and to continuously improve downtime and recovery work. The size and complexity of the hundreds of applications and thousands of servers, computers and devices currently managed require a deliberate sequence of events to occur with technical and operational staff to ensure the systems recover properly. The databases have valid and correct information and the integration among all these systems work as designed.

Many thanks for the immediate response of our teams to recover from this incident and restore power. The commitment, creativity and remarkable teamwork reflect true collaboration and dedication to our missions.