I’m kind of curious as to what people these days are doing on a UPS front, to keep systems running through power outages and provide a clean shutdown prior to batteries becoming exhausted.
It used to be common to see UPS systems sold to give desktop computer systems time to shut down cleanly.
The UPS market seems pretty stale to me. There have been changes over the past twenty years or so that I’d guess have caused some of that:
-
A move to filesystems structured so as to not risk corruption at the filesystem level from power loss at an arbitrary time.
-
Many people using laptops. Doesn’t change the situation much for servers, but I think that it reduces volume of the market that might want some kind of UPS.
I had expected that, with the drop in cost of lithium batteries and rise in tremendous rise in use of large batteries, that one would see new lithium-ion UPS units with large capacity.
But in practice, that doesn’t seem to be the case. UPS units are still around, but basically only provide a small amount of power, enough time to shut down. They aren’t normally geared up to keep systems running for hours.
There are lithium battery-based “home power backup” systems that provide loads of storage and automatic switching over to battery power if the mains power drops. However, these have some serious drawbacks that limit their use in a UPS role:
-
Some of these aren’t rated to switch over to battery power within a sharply-bounded amount of time, to avoid risk of momentary power interruption. For many devices, a momentary power interruption isn’t a huge deal, but for computers, it matters. I understand that on the order of 10 ms is expected for reliable UPS use, to keep computer power supplies happy.
-
One thing that one would like from a UPS is a clean shutdown prior to the battery becoming exhausted. For that to be done, the UPS needs to report its current charge capacity, so that software on the system can predict remaining runtime before exhaustion. Network UPS Tools is a widely-used Linux UPS-interfacing software package that does this shutdown. But looking at its hardware support grid, there isn’t support for these power stations, and I suspect that if there were reasonable charge-level reporting support anywhere, there would be.
-
USB has device classes that permits charge-level reporting, and looking at the USB spec, that appears to be true of USB PD. I have wireless headphones, for example, that make use of this. However, as best I can tell from looking through the kernel source, Linux doesn’t provide a way to treat these as a
power_supply
-class device, the way laptops have aBAT0
,BAT1
, etc, which would let the OS provide a clean shutdown itself when the time-remaining drops to a critical level. And even though power stations typically provide USB charging, I have not been able to find any that actually report their charge level via that USB in such a fashion.
I can think of at least three viable ways to do this:
-
I’m sure that there are people who have rigged up some kind of ad hoc system off a full-blown grid-tie power system, with separate batteries, inverter, charge controller, etc. In that case, all one needs is a voltmeter linked to the batteries prior to the voltage-regulation stuff, knowing what battery type is involved, and one could give a capacity estimate. Doing this ad hoc is going to have some drawbacks that I’d hope that a vendor-provided battery management system wouldn’t, like having to calibrate to one’s batteries and not automatically dealing with battery aging.
-
Simply run a UPS and a “big-battery” lithium backup power station. Plug the UPS into the power station and the computer into the UPS. The UPS provides the rapid changeover time and provides the computer with a warning prior to shutdown. This uses systems that should work out-of-box, but doesn’t really seem ideal to me in that one’s buying extra hardware and doesn’t have a unified view of time remaining on the battery – the computer thinks that everything’s normal until the power station is drained and the UPS kicks on.
-
Some people use old laptops as servers. For those, you can already use the OS’s built-in power management to deal with laptop batteries. If you have a power station extending the runtime, great, though in that case, you run into the same “you don’t have a unified view of the laptop and power station battery charge” situation.
I’m pretty sure that people out there doing self-hosted servers have thought about this, and I’m curious as to what people out there are doing in terms of the options out there. Do you just not worry about it, given the fact that corruption at a filesystem level isn’t such a big deal? Do you just use a UPS for a handful of minutes prior to a clean shutdown, and not try to keep your systems running through longer power outages?
I also don’t know how resillient home Internet connections are in the presence of power outages, whether typical cable, fiber, and DSL connections remain functional from the telco’s standpoint. I know that cell towers typically have some sort of generator setup, as I’ve read about those in the past, and believe that I’ve read that they typically can run for at least several days without power even without technicians driving out. I don’t know to what degree that is also true of wired communications hardware. I’m curious as to what the experiences of people who have put their server and network hardware on some form of backup power is. If you keep your on-premises hardware powered, have you retained Internet connectivity in power outages that you’ve experienced?
Ive been looking at those anker solix battery systems they supposedly can be used as a ups but haven’t looked into how well and you can plug solar panels into them which was my original goal to reduce the power usage from the grid at least for part of the day
I have a UPS designed to run long enough to shutdown my server gracefully. I can manually turn it back on if the outage is extended.
If I was designing a critical service the ups would have enough runtime to switch over to some other power system (solar batteries, generator, second circuit, etc)
I don’t bother. The last power outage we’ve had was something like 10 years ago and lasted 30 min. Keeping my network up during that time was not a priority at all…
So I just put a surge protector in front of all my PCs and call it good.
Cyberpower 1500 AVR for me. Router, WiFi, and server run off of it. It has kept me going on the internet during power outages as well as ding voltage correction when we have under or overvolt power moments. And I use the cyberpower Linux app they provide to set server shut down time based on battery level. It has saved my equipment a few times. We have occasional wind induced power failures here (overhead power lines) and had a crow land on a transformer and explode which immediately triggered the UPS to clean the sudden blip. Also had an HP laptop power charger fuse itself internally and UPS detected the short and shut down all power. Without that UPS, the pack might have overheated if the panel breaker didn’t sense the short.
You probably don’t need a UPS especially if it is more than a small fraction of the budget.
Use a modern filesystem like ZFS and you will be fine.
Its good for keeping your firewall or other services running, like homeassistant, during powered outages. But UPS also corrects power when it is dirty and protects your equipment.
I don’t remember the last time we had a power outage. I think it happened once like 10 years ago and lasted 30 min or so. The time before that was like 25 years ago, and I honestly don’t remember the details.
If we lose power, my network will be the last thing I’m concerned about…
Yeah, that’s probably a fair answer for some folks and systems. Depends on what your system is doing, your risk tolerance, and what money you’re willing to put into the thing.
If I didn’t have a backup system in place, say, that’d be much higher on my list than trying to ensure power isn’t cut off unexpectedly during power outages.
Yup, it’s a process of continually improving.
Battery backup is pretty far down my personal list of priorities, but it may be at the top for someone else.
You are lucky. Underground cables? We are West Coast of Canada, generally fine all summer, but November is windy season and maybe once a year it is out for a few hours when trees have taken out power lines. Its the undervolt or overvolt that people don’t notice unless they monitor it. My UPS sometimes kicks in for a minute or two, and the report shows 100v instead of 110v, or sometimes a surge of 129 for a brief period. 15 years ago I never cared, till I was doing workfrome home design contracting, and lost a whole days work to power outage and corrupted files. The UPS became cheap insurance.
We have underground cables in the neighborhood, and above ground cables along the main road. The city does a good job trimming the trees, so branches falling on lines isn’t really a thing. We’re also a desert climate, so there aren’t that many trees anyway.
That said, I grew up in the PNW (near Seattle) and we only lost power 2-3 times, though the outage was often for a few hours or even all night.
100v instead of 110v, or sometimes a surge of 129 for a brief period
Wow, that’s a crazy spike! I got a beefy surge protector, so I think I’m good, but I could see a UPS being worthwhile if your data is mission critical. Mine is just video games and some coding projects (which are backed up elsewhere anyway).
I agree that it’s less-critical than it was at one point. Any modern filesystem, including ext4 and btrfs, isn’t at risk of filesystem-level corruption, and a DBMS like PostgreSQL or MySQL should handle it at an application level. That being said, there is still other software out there that may take issue with being interrupted. Doing an
apt
upgrade is not guaranteed to handle power loss cleanly, for example. And I’m not too sanguine about hardware not being bricked if I lose power during anfwupd
updating the firmware on attached hardware. Maybe a given piece of hardware has a safe, atomic upgrade procedure…and maybe it doesn’t.That does also mean, if there’s no power backup at all, that one won’t have the system available for the duration of the outage. That may be no big deal, or might be a real pain.
I’ve never had an issue personally. Actually repeated power loss is one of my tests when I make changes to my setup. Repeated power loss quickly reveals issues with startup and fail migration.
I think you have managed to skirt around and discount the answer to your question. UPSs are not intended to be anything but a very short term power source. They are there to provide emergency power long enough to either shut down cleanly or migrate to another long term source of power. In your example with the “home power backup”, you have that and a UPS. The UPS smooths out the power and covers the few milliseconds during the transition. Even large datacenter UPS installations operate in this fashion. Datacenters would have a secondary source of power, typically diesel generators. The UPS batteries will be sized to run the site for a few minutes - enough time to get their generators up and running and the load transferred to them.
Yeah, I listed it as one possibility, maybe the best I can think of, but also why I’ve got some issues with that route, why it wouldn’t be my preferred route. Maybe it is the best generally available right now.
The “just use a UPS plus a second system” route makes a lot of sense with diesel generator systems, because there the hardware physically cannot come up to speed in time. A generator cannot start in 10ms, so you need a flywheel or battery or some other kind of energy-storage system in place to bridge the gap…but that shouldn’t be a fundamental constraint on those home large-battery backup systems. They don’t have to be equipped with an inverter able to come online in 10ms…but they could. In the generator scenario, it’s simply not an option.
I’d like to, if possible, have the computer have a “unified” view of all of the backing storage systems. In the generator case, the “time remaining” is a function of the fuel in the tank, and I’m pretty sure that it’s not uncommon for someone to be able to have some kind of secondary storage that couldn’t be measured; I remember reading about a New Orleans employee in Hurricane Katrina that stayed behind to keep the datacenter functioning mostly hauling drums of diesel up the stairs to the generator. But that’s not really a fundamental issue with those battery backup systems, not unless someone is planning on hauling more batteries in.
If one gets a UPS and then backs it with a battery backup system, then there are two sets of batteries — one often lead-acid, with a shorter lifespan — and multiple inverters and battery charge controllers in multiple layers in the system. That’s not the end of the world, a “throw some extra money at it” issue, but one is having to get redundant hardware.
RefurbUPS, new batteries every 3 years (i have handed over all my old ones to a local club and most of them are still going strong). I have a rack so i bought a rack unit. No rack might be cheaper even. Don’t cheap out though.
I have been doing the same, buying the batteries off brand individually is a lot cheaper then OEM labeled/packaged.
Though what are your thought’s on a UPS lifespan not taking in to account the batteries themselves?
I myself have three APC BR1500MS2
Honestly no idea. I use Home Assistant and just keep an eye on the stats. It’s in an unheated garage that sees humidity of 40-60% and temps from 50F to 80f and it’s just chugged along.
Personally, I just have a couple of cheap CyberPower UPSs for my servers. I know I know, but I’m waiting for them to get old and die before I replace them with something better. My modem, router, and primary WiFi AP are on a custom LiFePO4-based UPS that I designed and built, because I felt like it. It’ll keep them running for around 10 hours, long past everything else in the house has shut down.
I’ll add one other point that might affect people running low-power servers, which I believe some people here are running for low-compute-load stuff like home automation: my past experience is that low-end, low power computers often have (inexpensive) power supplies that are especially intolerant of wall power issues. I have had multiple consumer broadband routers and switches that have gotten into a wonky, manual-reboot-requiring state after brownouts or power loss, even when other computers in the house continued to function without issue. I would guess that Raspberry Pi-class machines might have power supplies vulnerable to this. I suppose that for devices with standard barrel connectors and voltage levels, one could probably find a more-expensive power supply that can handle dirtier power.
If you run some form of backup power system that powers them, have you had issues with Raspberry Pis or consumer internet routers after power outages?
I’m running an APC Smart UPS 1500 (APC-SUA1500R2) and I have never had any issues with pis. I’ve only used the 2/3 gens and a pizero.