1 reply to this topic

[Qwerty]

CBn was down for a little while today, as the server had a power spike which caused the site to be down for a while.

Sorry if this caused any inconveniences, but it was out of our hands on this issue.

CBn is back up and running now!

[The Admiral]

Thanks Devin for getting this post up.

This is the release I received from our server provider:

Earlier today, we experienced a very substantial power spike that damaged one of our UPs units at DC#1. As a result of this, which was actually multiple high voltage spikes, the charging and power unit in one UPS failed. Upon failure, this UPS went into emergency bypass mode, notified our NOC, and left it on core utility power.

Some minutes later, we were affected by approximately 4 brown outs that caused our PDU units on this UPS to shut down and re-cycle their power to prevent damage to the computers and equipment connected. These units worked as designed.

Our UPS technicians arrived on site within 35 minutes along with ALL of our core management team including Randy Williams, Patrick Smith, and Jeff Lowenberg. At the same time, we diverted some call center personnel to the DC and called in additional off duty personnel.

We made a decision to transfer from utility power to generator power for that UPS to ensure that additional brown outs would not affect server operations again. This unfortunately, caused an additional power cycle for a small percentage of our customers servers as well as some of our core servers, including the ones that power this forum.

Each of these outages was less than a few seconds in duration.

The affected part has been identified and a replacement has been pulled from our cold spares on hand. Replacement should be complete within the next few hours. When final testing is complete, we will switch back to primary utility power on that UPS which should be transparent to the customer.

In addition, we will be completing testing on all of our other UPS control systems to ensure that the power spike did not cause any damage to them.

Our personnel are working through all tickets as fast as possible. We appreciate your patience as we work as rapidly as possible to respond to all customer requests.