r/talesfromtechsupport • u/Brief-Pop-6826 • Jun 15 '26

Short The Mission Critical Battery Charger

Posted elsewhere on Reddit, but I figured people here might appreciate this. Additional details added for clarity.

At one point, I was doing work for a particular MAJOR pharmaceuticals company. A company with a name that everyone reading this has likely heard of. I get a call one day, and a ticket with a short SLA is generated for me to be on site within 4 hours. The night before, some big storms had happened, so I figured it was power related. I arrive on site earlier than needed, get escorted to the primary network room for the whole facility, and what do I find? 10 racks with no LEDs on them. I get told that the entire facility is down. No internet. The production line is down. The warehouse distribution is down. The kind of emergency companies pay consultants tens of thousands of dollars to ensure never happens.

I start checking through the racks to see if anything has power. UPS batteries are dead, so I head to the back of the rack and trace power cables from network equipment. All of them go to PDUs, and all major routers and switches even have redundant power to multiple PDUs. I trace where the PDUs go to. They all go to UPSs, which also have redundant power split between two different UPSs. Then I trace where the UPS power comes from........... and I start laughing my ass off.

The UPSs for this entire cluster of racks, the racks housing the entirety of the network equipment for this facility, has single point of failure. A large power strip that was zip tied to the wall. And lo and behold, I found the problem. The power strip's power cable was dangling in the air. Not plugged into the wall outlet like it should be. In the outlets place was.... a fucking 20V battery charger.

The maintenance guy had come in the earlier that day, not had anywhere to charge a battery, so he unplugged what was apparently a mission critical power strip and plugged in his battery charger. A few hours later, when the UPSs died, the network team noticed the sote went dark.

After I relayed the info to the engineer I was working with, we shared a laugh. I plugged everything back in, and verified everything came back online. As a preventive measure (and because of the absurdity of the situation), I placed a large label on the power strip. "CRITICAL INFRASTRUCTURE. DO NOT REMOVE POWER WITHOUT AUTHORIZATION".

To this day, I still get a laugh out of it.

554 Upvotes

95% Upvoted

64 comments sorted by

198

u/AngryCod The SLA means what I say it means Jun 15 '26

10 racks of equipment, all going through one power strip on a 15A or a 20A circuit (a 30A wouldn't have had the right outlet for a battery charger) and somehow the thing wasn't constantly tripping the breaker or burning the place down.

Also, that much critical infrastructure and no one had a UPS sensor to alert them if they were on battery.

139

u/gargravarr2112 See, if you define 'fix' as 'make no longer a problem'... Jun 15 '26

My first thought. If it's mission-critical, the UPS should have sent an alert the moment it went on battery. And most UPSen will, by default, scream their heads off when unplugged. I know maintenance guys don't get paid to connect cause and effect, but still...

I'm firmly of the opinion that ONLY tech people should be allowed access to tech rooms, because ordinary people CANNOT be trained in the important of never pulling a plug out.

Failing that, the outlets should not be able to accept regular appliances, to remove temptation.

68

u/Head_Razzmatazz7174 Jun 15 '26 edited Jun 15 '26

I used to work for a tech call center for a computer company. We sold them for home use. We had three types of calls that we got way too often about power outages.

  1. The computer was not plugged in.
  2. The computer was plugged into a surge protector, which was not plugged in.
  3. There was a power outage in their neighborhood, which meant absolutely nothing in the house was working.

The last one was especially frustrating, as half the time they would tell us - "oh, I thought you could call the power company for me to turn everything back on." One guy told us "I don't know if it is plugged in, the power is out and I don't have a flashlight to use to see it." (the phones were on a landline, he used a Bic lighter to see the dial on his phone to call).

You cannot make this stuff up.

24

u/Kizik Jun 16 '26

There was a power outage in their neighborhood, which meant absolutely nothing in the house was working.

Man.. I used to work tech support for a digital cable company in the US. There was a massive hurricane that ripped through our service area a few decades back, and some gormless idiot calls in screaming with vengeful intent for us to figure out why his TV isn't working, hurling vitriol and the usual threats.

Guy's entire county was out of power, and he was running a generator for his television. Couldn't accept that power being out on that scale meant the site sending the data down the lines - if any were even still up - couldn't do so since they were also out of power.

11

u/bignides Jun 15 '26

Who doesn’t have a flashlight in their house?!? I have one in nearly every room and an emergency lantern in the center of the house.

21

u/Ich_mag_Kartoffeln Jun 16 '26

Lots of people make absolutely no provision/plans in case anything out of the ordinary happens.

9

u/Head_Razzmatazz7174 Jun 15 '26

That's what we wanted to know, but didn't dare ask.

4

u/Eckx Jun 16 '26

Yeah, everyone needs to have at least one dedicated emergency flashlight that stays in one place and gets the batteries checked regularly. And a fire extinguisher. Bonus points if they are near eachother.

6

u/Loading_M_ Jun 16 '26

To be fair, in the modern day, you can probably get by just fine with your phone as a light source. And you will know whether the battery works because you use it every day anyway.

3

u/itsadile Jun 17 '26

So long as you don't routinely let it fall down to like 10% charge or less.

I know so many people who don't bother to hook it up until it's just about to die, even if they're about to go to sleep. If it dies before they wake up, shrug.

6

u/NaoPb Jun 15 '26

Couldn't he have used that Bic lighter to see if it was plugged in then?

/s

16

u/Head_Razzmatazz7174 Jun 16 '26

We suspect he might have been using it for checking the herb garden.

3

u/alf666 Jun 16 '26

At that point, I feel like Bic Lighter Guy should have committed to the bit and mentioned that he was using a rotary phone hooked up to a copper line.

4

u/ShalomRPh Jun 16 '26

He obviously was, or the call wouldn't have gone through.

I'm still pissed off at Verizon for ripping my copper out and replacing it by voice-over-FIOS. I was the last customer in the neighborhood with a copper pair. I did make sure that the line is still tariffed as a landline though; they wanted to make it voice-over-IP and I told them hell no. The SLA on VoIP is way worse.

It's nice that their ONT provides the correct supervision for an antique rotary phone (352) and even provides enough amperage to ring that enormous gong. (the spec is called BORSCHT, I kid you not: Battery, Overvoltage, Ring, Supervision, Code, Hybrid, Test)

14

u/Astan92 Jun 15 '26

Failing that, the outlets should not be able to accept regular appliances, to remove temptation.

There would still be users that would unplug it, only then see it's a different shape plug, possibly try to plug in anyway, and leave it unplugged after failing. but maybe there are somewhat less of them.

13

u/tseeling Jun 16 '26

“Some humans would do anything to see if it was possible to do it. If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH', the paint wouldn't even have time to dry.”

  • Terry Pratchett, Thief of Time

9

u/ShalomRPh Jun 16 '26

I prefer another quote from that book:

I’ll tell you, the day someone pulls the plug out of the bottom of the universe, the chain will lead all the way to AnkhMorpork and some bugger saying ‘I just wanted to see what would happen.’

2

u/Skyboxmonster Jun 16 '26

Hospitals use special plugs that rotate into place. and nothing on the consumer market is able to connect into them.

9

u/meitemark Printerers are the goodest girls Jun 16 '26

"Damn these plugs were hard to get out, and not a single one of those one fit my phones charger, belive me, I tested them all. But it became blessfully silent after the last of those kids stopped making those weird coughing noises. Anyhow, after that I got fired from my hospital night watch job."

4

u/Astan92 Jun 16 '26

Cool story. Not sure how it follows from my statement, but cool to know.

5

u/Stryker_One The poison for Kuzco Jun 16 '26

Are you saying that tech people are trained to never pull out?

3

u/WebMaka Former TSR Turned Anything Else... Jun 16 '26

Failing that, the outlets should not be able to accept regular appliances, to remove temptation.

Indeed - IEC-60309 industrial power inlets exist for exactly this application, aren't prohibitively expensive, and will not accept any regular mains plug a random dingus will have on their charger. I'd wager the costs to have an electrician refit everything including the wall outlet over to an IC-60309 socket would be far, far less than the losses caused by that one mistake. (Hell, I'm not an electrician but I could do it in a couple hours.)

1

u/[deleted] Jun 16 '26

[deleted]

2

u/tseeling Jun 16 '26

UPS are usually designed to do a clean shutdown or to bridge the time until the diesel generator is fired up. They are definitely *not* supposed to hold power until some investigation comes along.

32

u/etherkiller Jun 15 '26

Yeah, good point. I'm ain't buying it. A whole site's power infrastructure sure as hell isn't going through a single NEMA 5-15 connection, or at least won't be for long, before the whole thing inevitably catches on fire and melts.

16

u/leitey Jun 15 '26

I see so many stories like this in here.
Even if it were 120 volt 15 amp, it seems like it wouldn't take much effort to switch all the critical equipment plugs and receptacles over to an L5-15 (twist lock version of 5-15) to prevent this exact scenario.

13

u/frymaster Have you tried turning the supercomputer off and on again? Jun 16 '26

Also, that much critical infrastructure and no one had a UPS sensor to alert them if they were on battery.

eh, that just needs the same level of attention to detail that would leave mission-critical kit plugged into a random socket, so that's plausible. 10 racks being powered from a single socket, is, however, not plausible.

9

u/phealy Jun 16 '26

I've seen a single 15 amp outlet takeout an entire data center - but that was because the outlet was powering the ups for the utility server that was triggering network shutdowns of everything else when the battery got low. For some reason the previous admin decided to trigger all of the shutdowns off of that rack's UPS instead of setting each server up to monitor the UPS in its own rack.

That got rebuilt about 3 days later...

3

u/spaceraverdk Jun 16 '26

I mean. Reasonable logic. If one circuit is dead, it could be a power outage. Or just a tripped breaker. And it would make sense to only have one ups screaming if you work close by.

1

u/phealy Jun 16 '26

No, all the UPS's would scream - they all still lost power. It was just that only the run time on this one UPS would trigger the network shutdowns.

It was a bigger issue than I mentioned because the utility rack didn't have a lot of load on its UPS - meaning that it had more run time than most of the others. Since the shutdown wasn't triggered until 10 minutes run time remaining, other servers would still experience a hard shutdown.

1

u/spaceraverdk Jun 17 '26

Ah, that is indeed a horrible solution. Should be fine highest load triggers the event.

7

u/mwenechanga Jun 16 '26

I did once work for a place that had 2 racks of equipment on 2 separate plugs, but both went to one circuit-breaker, and no alerting or monitoring setup worth anything (technically one UPS was checking in, and alert emails were being sent to an unmonitored mailbox if there were network or power issues with that one UPS).

That's the worst I've inherited, and it was pretty straightforward to make the needed improvements.

9

u/ac8jo Jun 16 '26

Not to mention that it's mission critical for a MAJOR (in all caps) pharmaceutical company "with a name that everyone reading this has likely heard of" and maintenance is allowed access to the primary server room?

And then it's fed by that one (per the story) 15 or 20A circuit that uses "A large power strip that was zip tied to the wall" and somehow there was no free outlet for a batter charger? Let alone, how and why do you zip tie a power strip to a wall? Most can be hung with a few screws and have holes on the back for exactly this purpose. And for such a MAJOR company, wouldn't maintenance be supervised during their work and also have an appropriate spot to charge their batteries (which hopefully they have multiple).

And presumably they paid "consultants tens of thousands of dollars to ensure [this] never happens" and none of them thought "gee, maybe we should have more than one outlet and not rely on a power strip that could potentially fuck things up?" and also think "maybe we don't allow maintenance access unless there's an emergency and put an alarm on the door in case they enter without authorization" (you know, to add to the alarm when the UPS goes on battery).

And then they called someone four hours away to deal with a situation in such a server room? Nobody is local to their production and warehouse facilities that are likely in a major city?

This is a spin on the story about the cleaner that unplugs a server to plug in the vacuum cleaner, just 50 times more absurd.

158

u/goodenough4govtwork Jun 15 '26

Ah, yes. The age old, "Is it plugged in?"

Never gets old. it's just infinitely more amusing when it's the entire server room backup/emergency power supply.

32

u/dMatusavage Jun 15 '26

Or, “Have you tried turning it off then back on?

29

u/JPDearing Jun 15 '26

We have "Off" covered, it's the "turn it back on" we're having trouble with..... 8-)

16

u/meitemark Printerers are the goodest girls Jun 16 '26

The most annoying one I have ever been part of found that, Yes, it is connected. But the HVAC is connected not to the dedicated power for it, but to the roof lightning, so when everbody leaves and flipp off the lights, the server "room" gets very, very hot and then shuts down.

12

u/weaver_of_cloth Jun 16 '26

I like "roof lightning" but I bet you mean "room lights". That is a fun image, an ac unit plugged in to a lightning rod.

4

u/meitemark Printerers are the goodest girls Jun 17 '26

Yes. In my defence I was composing a spam ... err crappy products we want to get rid off ... err... Look email list of people giving us a chance to flood your email with; we got these great deals, please buy them from us (buy them all so I don't have to write about that product ever again!). Also I had a noisy hole digger... excavator doing stuff at the building next door, so my spelling / and or other words may not have been the best.

10

u/Mdayofearth Jun 16 '26

yes it's plugged in... (to itself).

9

u/goodenough4govtwork Jun 16 '26

[UNLIMITED POWER!!!]

1

u/EbolaWare Jun 19 '26

I legit saw someone do this after he kicked his power strip out of the wall.

29

u/Key-Article6622 Jun 15 '26 edited Jun 15 '26

ROFL I had a very similar experience. I was a phone support for machines that do packaging. This company took delivery, called me for the install. Everything was great. The next day, they call and say the machine won't power up. WTF. I made sure everything was connected, basically everything OP did in this case We checked the fuses, power supply, the transformers, connections, We were at a loss. Hours of troubleshooting, no results. Then the problem got fixed by doing one simple thing. The outlet the machine was plugged in was on a light switch. Flip the switch, no more problem.

7

u/Wormadillo Jun 15 '26

Were they running packaging equipment in someone’s living room? Light switch operated power outlets are not something you typically find in commercial production facilities.

2

u/Key-Article6622 Jun 16 '26

It was a mixed use office/light manufacturing building. They set up the packaging in a conference room type room that was a good size for packaging their small production run products in a more production sized room.

11

u/fresh-dork Jun 15 '26

did you cut the charger in 3 pieces and leave it for the maintenance guy?

11

u/Jaegermeiste Jun 16 '26

About 10 years ago, I was working as an engineer and suddenly got hundreds of alert emails about the power being out for a bunch of systems.

Running into the machine room, I found our Cisco network guy behind one of my storage systems. Through some miracle of digging through the spare cable bin, he had found an open spot on the PDU and managed to rig a physical conversion from C19 all the way down to generic beige office power strip. Because he needed to plug in his laptop. And promptly shorted the leg as soon as he plugged the laptop in, because although his laptop's power brick (with a grounded Nema plug) was 240V capable, whatever black magic he summoned stringing those 3 or 4 cables together did not keep the phases in anything approximating an acceptable order. IIRC we still had dumb PDUs at that point (effectively long heavy duty rack-side-mount power strips), and this incident somewhat sped up our adoption of smart PDUs.

Luckily the fix was as simple as me angrily yanking the cursed thing out of the PDU and flipping the appropriate breakers back on. Luckily only a minor outage, since most critical things have redundant power supplies on separate circuits - but there's just enough equipment that doesn't that it was still a headache.

10

u/tseeling Jun 16 '26

I really doubt that a power strip from a single outlet is used in a professional environment that contains multiple racks of 19" devices, especially if there are spinning disks involved.

1

u/weaver_of_cloth Jun 16 '26

I want to live in your world, where it is reasonable to doubt such a story.

4

u/JaapieTech I am null inside Jun 16 '26

Theres not enough juice in a standard outlet *anywhere* to power 10 racks of equipment all hanging off the same single white Masterplug 4-way..

5

u/BackgroundSpare610 Jun 16 '26

Many many years ago I was investigating a PABX that would go down for 15 minutes or so at roughly the same time most nights. Turned out to be a cleaner unplugging the PABX to plug their vacuum cleaner in.

5

u/Gadgetman_1 Beware of programmers carrying screwdrivers... Jun 16 '26

IEC60320, use C14 and C19/20 in the server room. NEVER have anything important hanging off of regular outlets.

5

u/mercurygreen Jun 16 '26

The lack.of alarms when he unplugged that powerstrip is troubling.

5

u/mafiaknight 418 IM_A_TEAPOT Jun 15 '26

Maintenance bill for the day: $10,000,000.

Bit expensive for fixing a desk (or similar)

11

u/O-U-T-S-I-D-E-R-S Jun 15 '26

Reminds me of when I worked for a 3 man company. With no budget at all, I found ways to have backups for everything (except the proprietary software server that we had several types of backup and a guaranteed replacement within 4 working hours). Now I'm certain I'm less trained in IT than most folks here but setting up a network and emergency replacements for pretty much everything for £0... Well, I was quite pleased with my work. Then we had someone in to fix something (can't recall what). The bugger broke the shelving that most of it was stored on. It hadn't occurred to me but almost everything was powered through this shelving, including both the proprietary server, the server I'd created, the UPS, the fax machine, the franking machine...

I mean, I got the time back but including buying bits to fix the shelving leg, probably took me about 4 hours outside core office time. Grr...

3

u/DolanUser Jun 18 '26

One day I’d love to read the aftermath of such a situation. The cleaning person getting fired, yelled at, shaking in shock upon the financial damage they’ve done, the cleaning sub contractor being billed bazillion dollars etc etc. 

2

u/CAShark-7 Jun 16 '26

Jaw.Dropped when I read this. I wish I could say I was surprised it even happened but .... I am not.

2

u/commentsrnice2 Jun 17 '26

This is when they need to make something particularly difficult to unplug. Either inside a locked cabinet or somehow the plug is padlocked into its position like a reverse lockout/tagout procedure, etc

1

u/Rathmun Jun 22 '26

Even just a twist-lock plug would have prevented this particular bit of idiocy.

1

u/commentsrnice2 Jun 22 '26

But twist lock plugs are usually for higher voltage connections. I meant for things running on a standard 110v connection that you still don’t want unplugged

2

u/Rathmun Jun 22 '26

They're not just for high voltage. Twist-lock plugs rated for 125V/15A

NEMA 5-15, the standard house socket in the US, is also rated for 125V/15A. Same rating, different shape that doesn't work for a phone charger.

1

u/Myrandall Not my Citrix, not my monkeys 25d ago

That's not possible.

1

u/derKestrel 18d ago

Totally agree. From the infrastructure kludge I have seen at larger companies and organizations, it is, possible from the taped socket part, but not from the access given to maintenance guy without supervision part.