Fun with power backup

One more tangent while we’re reminiscing about the Good Old Days. There’s a major power outage going on in San Francisco right about now, which may be the result of human error, to put it nicely :-)

It’s funny how this stuff still happens despite oaths to the contrary by data center providers. But it does!

A few years ago, my company had a lot of equipment co-located at a Switch & Data facility. They are (were?) a nationally recognized, relatively well respected data center company, and the location we were at had all of the modern trimmings: nice cages, dual utility power, a monstrous diesel generator, tons of A/C, controlled access, yada yada…

One day, predictably enough, both utility power circuits went down. The batteries kicked in and the diesel started to spin up. It got halfway up to operating RPM, but there it stayed. The local Switch team converged on the site and tried to get into diesel room. Only – it was locked!

But it wasn’t locked! You see, when the gigantor humungous diesel fired up, it started sucking tons of air. The wasn’t sufficiently ventilated, as it turned out, although nobody ever would have known, because they always tested with the door open (since they were in there watching anyway).

So there they were, with the diesel running way under operating RPM because it couldn’t get enough air, and they couldn’t get in to the room to diagnose it, because the door opened out and it was therefore vacuumed shut!

They wound up having to break the door down (not easy with a heavy commercial door and jamb), and hours later, the site came back online.

Data centers are a pain. Try to have at least three.

One Response to “Fun with power backup”

  1. Rob says:

    They were using a heavy duty diesel generator inside without proper ventilation? What sort of cowboy installed it? Likely had they tried to test the generator with the door shut, they would have suffocated — here in the UK, a couple of guys have died over the past week because they used a small diesel generator inside to try to remove flood water from their cellar.

Leave a Reply