Rich's Tongfamily (Richtong.net add-on)

OK, we just had an interesting network event. Last week, we got a backup cellular connection for our home network since we are doing Work From Home (WFH) so much. Well it turns out it worked so well that we are seeing micro outages with Comcast Xfinity for a minute or so. I think these are probably related to the automatic pinging to ‘ping.ubnt.com’ which is actually routed to the Google DNS services at 8.8.8.8

But what happened today is that we went into failover mode about 1PM and then I had a very low 2GB data plan with US Mobile for $15 a month and when that runs out, the network doesn’t work anymore. It was confusing because, the Comcast modem was reporting that there was a connection. All the lights were green. The basic idea is that when this happens, there’s a power plug which reboots the Comcast modem and you are supposed to be back.

The second strange thing was that the local Unifi Console on the Unifi Dream Machine Pro wasn’t responsive. It just showed no traffic to either the main WAN or the backup WAN2. So here is what I did:

1. Manually rebooted the Unifi Dream Machine Pro, the SB8200 Cable modem and service was restored, but the UniFi console didn’t come up at all.
2. Thanks to the miracles of CannTech and Reddit, I was able to find the right command to ssh into the UniFi Dream machine Pro and then run /etc/init.d/S95unifios restart which means that it will reboot the Webos. You have to enable SSH as root into the UDM Pro, but it does seem like that is the only option when the interface goes down.
3. It turns out that I was not getting notifications that there was a WAN to WAN2 failover, you have to get to this at the UniFi console at https://10.0.1.1/network/default/settings/system or whatever the address is of your UniFi console and then in Network > System, you will see that notifications at the top and I choose Configure Custom Settings to basically turn it all on and have it emailed to you. In this case strangely, the log didn’t show a cutover to WAN2, so I’m guessing I hit some sort of bug there.
4. Cable modem reboot is something that should work, but I can’t see any notifications for when it does this, but this little $29 gizmo is supposed to have the a restart if it detects it is down, but I can’t find any notification or figure out if it is really working. It shows up in the console, but how do you configure it. I’m not sure. Evan McCann says its unreliable, but I can’t even figure out it if works 🙂 Because although I have events turns on in the notifications section I don’t see it is ever turned on.
5. Turn off network intrusions and Network client connection change. This is on by default and really noisy, so you should connect your phone with the UniFi application so you can see alerts that are happening, but make sure to turn off Network Client Connection Change in Advance notifications and Threat Detection and Prevention as these are too noisy in most environments and then make sure that Detailed notifications are turned on with your phone. You don’t want these critical things to appear in a Scheduled Summary on your iPhone. You change this in Settings > Notifications > UniFi Network > Immediate Delivery. It kind of shame there is no way to filter out all those network intrusions in the UniFi Notifications web page like you can with alerts. There are so many in most systems it drowns out the important ones like network connectivity lost.

The net of this at least for me UniFi seems work pretty well but I seem to have had a failover bug which caused me to fix my notifications and learn how to reboot the UniFi Console too.