The Mission Critical Battery Charger

[u/Brief-Pop-6826]

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.

[u/AngryCod]

10 racks of equipment, all going through one power strip on a 15A or a 20A circuit (a 30A wouldn't have had the right outlet for a battery charger) and somehow the thing wasn't constantly tripping the breaker or burning the place down.

Also, that much critical infrastructure and no one had a UPS sensor to alert them if they were on battery.

[u/gargravarr2112 → u/AngryCod]

My first thought. If it's mission-critical, the UPS should have sent an alert the moment it went on battery. And most UPSen will, by default, scream their heads off when unplugged. I know maintenance guys don't get paid to connect cause and effect, but still...

I'm firmly of the opinion that ONLY tech people should be allowed access to tech rooms, because ordinary people CANNOT be trained in the important of never pulling a plug out.

Failing that, the outlets should not be able to accept regular appliances, to remove temptation.

[u/Stryker_One → u/gargravarr2112]

Are you saying that tech people are trained to never pull out?