Do not Factory Reset your Unifi or suffer the consequences
Well, I'm an idiot, there is no doubt. I've been pushing the envelope of the system pretty unknowingly and I had not seen a comprehensive, here's how you bulletproof your system and what are the best practices. So first, what to do if you are an idiot and then how to keep it from happening ever again.
What to do if you lose your web interface, video, etc.
OK, the problem with networking is that things sort of degrade and you have no idea why, so in my case, here were the symptoms:
- Unifi Protect which has video cameras stopped showing any of the videos except what was live, you could go through the timeline, but clicking led to nothing.
- One of our cameras kept going on and offline at random intervals. It is on a long cable run with POE injectors on each side and was only showing 10Mbps. The injectors can do up to 100Mbps, but this is the longest run. The cables are buried so it's easier to inject power and network than to do something else.
- The web interface at https://unifi.ui.com was hanging all on the current v1.9.3
So being an idiot, I thought a quick reboot of the Unifi system would work. I had the thing on auto backup, from somewhere, so what could go wrong, well here is what happened:
- I couldn't see anywhere in the user interface how to do a restart, apparently, you have to ssh into the controller (which I've avoided doing for a long time), but I just unplugged it. And on reboot, the network worked fine, but the message was 'UDM could not startup' on the tiny little 1.3" display.
- There is URL to go to figure out how to get it into Emergency Recovery mode. But basically, you have to power on the system by plugging it in (there is no power button at all on it), it's a little tricky, you hold the reset in with a paper clip on the front right while plugging it in at the rear left.
- Then you have to plug into port 1 of the Unifi Dream Machine Pro and then set a static IP address in the 192.168.100.10 range and you then connect to the magic port and there are a few buttons. The first one is "Factory Reset" and there are others which are restored from backup. So I was an idiot and just hit the reset
Why a factory reset is a bad idea with Unifi it is a real operating system
As an aside the main and hard thing to understand is that internally, Unifi has in their Unifi Dream Machine and Unifi Dream Machine Pro rearchitected how they work. Now they are running a Linux variant with a single Podman container called unifi-os. So what you have are different versions for that container called the Unifi OS and then there are applications that run on top, so Network is the most used, but they also have Access for their card key system and Talk for their VOIP system. So when you are doing configurations, you are configuring both the OS and the applications usually Network. Then in each device actually runs Busybox or some cut down operating system as well. You can SSH into each device and there is a MongoDB database inside the Network application that keeps track of things. So as you can see you are using a real system and if you just factory reset it, it can be disaster without some preparation (as I painfully learned).
So here is what broke
What a mistake! I did this and when I rebooted, I did find a brand new v1.8.6 but:
- None of the networks worked because I was on 10.0.1.1/24 as a network but the default is 192.168.1.1/24 so the access points could not connect.
- When I tried to use the SSH user name admin and password that I had stored, it didn't work. With Unifi, it showed all the APs and network devices and "unadopted". This is their term for being able to manage and use them. And when I tried "Advanced adopt" I discovered that this was incorrect. What has happened is that the system now has a randomly generated account and password for each AP. Probably done for security reasons but this means the only choice was a hard factory reset that requires physically pushing the reset in *every Unifi device*. Ugh, I have literally 20 of these devices (a router, 2 switched, 5 desktop switches, 5 access points, 4 cameras, and a telephone). Some of these are literally mounted three meters high and they all need to be taken down and reset.
- It gets worse though because the reset is actually pretty unreliable. One of the first things I did with the reimaged UDM Pro is that I switch the network back to 10.0.1.1/24 and that mean all those devices had the wrong IP address. I didn't think this would be a problem, but it really confused the system. Net, net, don't change from the default until the last minute.
- Also, I discovered that all of the automatic backups I had taken were deleted in the factory reset. You actually have to physically download the configuration files and back them manually. I had somehow thought this was going to work better. That at least the cloud would have backups, but you need to set this up yourself. There are tools like backifi.com that do this automatically to a cloud service I probably need to use. Although be aware that there are no OAuth tokens, so you have to give them a clear text super-admin password to deal with backups.
- Now, factory defaults do *not* change the firmware you have loaded, it just zaps the settings.
- But if you did this correctly, then with your cell phone, the included Bluetooth transmitter will detect and you can continue setup from your phone. I actually found that a factory reset failed, it would not complete the setup with v1.9.3, so I had to download the older v.1.8.6 and with that firmware, it could complete the setup. So I had to get back into Emergency Recovery Mode and then in the Firmware Update, it browses on your local computer and uploads it to the computer.
Resetting devices and readopting
So, Unifi is basically useless for this large scale operation because the ways that you reset each device are quite different, so if you end up here, here are some notes about it:
- Unifi APs. They all work pretty much the same way, you have to unplug them and then plug them back in. It's really important that you power up and only then push in the reset pin that is on the back with a paper clip. If you power it on while it is plugged in, you will put it into TFTP mode and it won't pair. Also, don't push in too hard. I pushed in too hard a while ago and broke the reset switch. If you do that there is no way to recover. Now, if you make them work properly if you have previously given them an IP address that is different than what the router is looking at. I made this mistake as well and what will happen is the things get discovered and you can try to adopt but it will fail. The light patterns are unintuitive but when you power up, it flashes every 0.5 seconds white when booting, and is you did the reset properly should go to a steady white. Note that for modern APs like the Unifi 6 LR, it has a Bluetooth transmitter, so if you have a cell phone near you, it will actually ask about setup.
- Unifi Enterprise Switches. OK, with these, you need to power them off from the tiny little screen on things like the 48-port and you do the same thing, turn them off and then press the reset button with a paper clip for five seconds. The cheaper devices like the 24-port switch use the same light pattern as the APs, but fancier 48-port ones have a tiny 1.3" touchscreen and it tells you its status. Now to get them all working again, there is not much configuration unless you have Link Aggregation set up. That is using two ports as one to get 2Gbe bandwidth. You need to go to the Unifi Controller into Devices then on the USW switches, to ports, select the port to aggregate, and then there is a confusing dialog that says port X and has an invisible field that tells you to want other port you should aggregate.
- 5-port USW Flex and USW Flex Mini. These are tiny little desktop switches and yes, they need to be reset. For the POE USW Flex, there is a button on the back. For the unpowered switches, there is a reset with a paper clip that you need to hold for 10 seconds.
- G4 Pro cameras. These are actually pretty simple if they are not too high, you unscrew the front part of the camera, and next to the Ethernet is a paper clip reset. You push that in while you are powering on (confused yet, there is no standard way to do this) for 10-20 seconds. It works when the ring around the camera lens flashes three colors, then it has the same pattern as the AP, blinking white and then solid white when ready to pair. As an aside when I took this apart, I saw the cause of the disconnect bugs, I had gotten silicone waterproofing into the ethernet so it was physically blocking some of the power pins.
- G4 Doorbell. OK, this is a bit of pain, but first, there is a cranky tab underneath the doorbell, you have to press it and then pry it from the bottom. It is tricky, I broke the tab 🙂 And then don't pull too hard on it since the two-wire connection is held with very weak spring clips. Then to reset, you push the button at the top back of the doorbell for 10 seconds and it will reset. For me, at least with this recovery, it seemed to work.
- Unifi Phone. Ok to do this, you have to unplug the phone and then hold the mute button on the handset when you plug it in. Then you need to start the Unifi Talk and set that up.
OK, how to build a more robust system
Well, the first thing is be more circumspect about taking updates:
- Make sure you have scheduled backups running. The really confusing thing about Unifi is that the default web interface you get does not have all the features of the classic interface. And the interface on the phone is a subset of that, so you will spend a lot of time figuring out whether to use the classic (and complicated!) or not. In this case, all the backup settings are classic and not new. So the easiest way to deal with this is to register your device in Unifi Cloud or unifi.ui.com. This will figure out how to easily access the web interface. Go there, select your network/Network/System Steggins and then there is a button New User Interface which you have to turn off. Then in this UI, it is in Settings > Backup > AutoBackup.
- Copy some of the backups to your Google Cloud or some storage system. And remember these backups just live on your controller, so you need to click on the Backup/Restore Download Backup and then pick the interval and then download and this will give the complete UNF file so the next time, you can restore from that. When you are done, you switch back to the new UI with Settings > User Interface > New User Interface and click that on.
- Now turn off the automated software updates, so you can control your own downtime. You only want to run an upgrade when you have a copy of the backup running. Confusingly this happens in two places in the New UI. For the non-controller, go to System Settings > Maintenance and turn off Automatic Firmware Upgrades.
- Then for the controller or router itself, browse up and there is something called OS Settings which is actually in the User Interface on the buttons below. Basically, in the system, the UDM Pro runs as the OS and then there is the Network, Protect, and other applications and each has a version. So v1.9.3 refers to the Unifi OS while there are other versions for the "apps". Also at the bottom is where you can set up users.
- So you need to get out of the network app but clicking on the top left button and will get you to a list of systems, click on your current network and you will be at the Portal page and click on System Settings at the bottom and then at the top level click on Advanced and you want to turn on SSH, so you can access the controller (separately from the other devices) and then set a unique SSH password and definitely allow Remote Access, so you can use the cloud to get to it. And here is the Power Off settings (I know weird right, it should be higher than in an optional hotlink).
- Then you click on Backup Configuration on 1.9.3, Theis store backs in their cloud under your UNifi ID. I've actually no idea how to restore from it, but having backups is pretty essential unless you enjoy spending a day resetting all your devices.
- Now in Updates, turn off auto-update for the Dream machine and all the applications. You definitely want to make sure you have a working system before doing these. NOte that when you do this, you have to do a manual update in OS > Updates and this will often fail mysteriously. On some machines, you get a countdown, but on others I'm finding the update silently fails. Argh!
- If you don't want to trust Unifi, you can also allow remote access to your machine with Dynamic DNS. Personally, this just seems like a security hole, but it's possible.
- After you after have finished close out of that and then click on Users and add a backup user so that if your account gets compromised you have a backup say Gmail identity to get in by clicking on Add User at the upper left and then adding a new Super Admin.
- You can also create a unique device ssh name and key so that you can SSH into each device. This is a completely different key from the controller so you need two sets of keys.
- If you are using the map feature, you also need to create a Google map API key so that you can see your site information. Then go to System Settings > Controller Configuration > Google Maps API Key although in the current release it doesn't work this way.
- And to make sure you get alerts, enable Network > System Settings > Controller Configuration > Mail Server and select Cloud email so you get alerts.
Record your Device User name and SSH password
OK, all that was important, but the most important is recording the device user name, this is buried in the Network application in Network > System Settings > Controller Configuration > Device SSH Authentication and stuff into 1Password someplace the random user name and Password for all the Unifi Devices on your machine.