A house call goes wrong

While looking after his church’s Ubiquity UniFi network plant, the Moocher scrapes his knuckles. A simple update and put guest policies on the various guest wireless networks.

A year ago a local managed network services contractor configured our USG, switches, and access points. The church played the role of contractor subbing some of the wiring, doing some ourselves, and hiring a network installer to dress out the rack and install the service outlet plates. A second contractor configured the church purchased network core.

A year later, we discover that one of our contractors had left the guest WiFi networks open to the main LAN from whence the controller could be configured. So change that right? Yes, but not from the WiFi. All hell broke loose. The hot wash is after the break.

References

  1. https://dl.ubnt.com/guides/UniFi/UniFi_Controller_V5_UG.pdf

The Moocher’s Tale of Woe

The Moocher’s home network has a newer Cloud Key Gen2+ that shares network configuration and video recording duties. The church has the older primordial Cloud Key with a uSD card installed for backups (our savior).

In the process of trying to configure the WiFi networks for the guests, the Moocher inadvertently altered the configuration of the main network and staff WiFi where we have our audio mixer iPad interface and our EcoBee thermostats, VoIP phones and PCs. Things went wrong mid-afternoon Monday after our administrator had left for the day.

Admin returns to work on Tuesday and can’t reach Quick Books or Icon Church so all hell broke loose. On Monday, the Moocher struggled in vain to revive the confused Cloud Key by restoring a backup from the auto-save library on the controller. It kept going wrong. One laptop was WiFi only but the WiFi was busted. The second laptop had an Ethernet connection but was Win10 and the interface could not be set to the Cloud Key’s no DHCP network. (192.168.1.30 after hard reset). So the Moocher took the Cloud Key home to revive it. This took less time than feeding the greyhounds. Life is good right?

On Tuesday the Moocher assures panicked Admin that all will be right in an hour. Moocher drives to church, plugs in the Cloud Key and metaphorical smoke comes out. DHCP, and routes were still wrong. Cloud Key was waking, taking a DHCP server timeout, and coming up on the Class C network rather than our Class A slice.

Moocher takes Cloud Key home and resuscitates it yet again. This time he changes the no-DHCP fallback to be on our church network at the Cloud Key’s normal spot at the top of its slice. Cloud Key starts. Main network again has Internet service, WiFi again works.

Hot Wash

After this sort of debacle, it is good to have a hot wash to reflect on what went right, what went wrong, and how to be better in the future.

What Went Wrong

  • Never ever do network admin from a WiFi only laptop. If you break the WiFi, you are hosed.
  • Never ever do network admin from a track pad only laptop if you can’t keep your mitts off the track pad. Stray track pad touches and graphical administrative interfaces don’t mix.
  • Don’t save network configuration settings backups on the WiFi laptop. When the WiFi is borked, you won’t be able to restore that backup.
  • Always bring a keyboard and mouse and possibly a Raspberry Pi for break glass in case of emergency recovery hosting.
  • Our church had a mongrel collection of desktops but nobody knew the alternate boot procedures to bring Linux up from a USB nub.
  • Church had no HDMI monitors, just a cheap TV masquerading as one. There were not enough pixels to run a modern UI. Pixels are important so data entry boxes work as the designer intended.
  • No church computer had an OS on it that could have its Ethernet interface manually configured to the Cloud Key’s no-DHCP network.
  • After reset Cloud Key’s no-DHCP fallback was off the church network so it couldn’t be reached in place.
  • Management credentials are not held in escrow. If the one parishioner who knows them goes missing, church is up the proverbial creek without a paddle as passwords are good.

What Went Right

  • We had a spare Switch-8-150W that we could use to make a bootstrap network. Sadly, we had no computer to connect to it.
  • Our Cloud Key was saving monthly auto-backups internally so we could revive it easily.
  • Taking the Cloud Key home and bringing it up on the Study Switch, it was easily found, logged in, and revived.
  • The one parishioner had a good record of church passwords in 1Password.
  • On the second go, the Cloud Key’s fallback network was manually configured to its designed spot on the church network.

Church network came up and Administrator recovered from swoon.

Long Term Actions

  • Have a second go at creating the guest networks and hooking up the 2 guest WiFi groups to them. After the new financial year is started.
  • Purchase at least 1 high resolution HDMI monitor for sysadmin work. A staff member, say Comm Coordinator can use the monitor when it is not needed for sysadmin tasks.
  • Purchase at least 1 Raspberry Pi 4B starter kit so we have a simply managed computer for sysadmin work.
  • Purchase at least one high quality Anker USB 3 hub that can power larger USB storage devices. Older computers don’t have the juice. Pi’s don’t have the juice.
  • Install a switch at the AV desk in the back of the sanctuary. This is a good place to do sysadmin work as there is lots of desk space to set up. We have a switch for this purpose.
  • Install a switch for the stage Ethernet. we have the wire run. We have the switch. we’ve just not mounted it and cabled it up.
  • Consider managed network service with a local admin shop. Typically, this has cost more than the budget would bear so we’ve been DIY.
  • Find a second congregant to be a sysadmin and subscribe to 1Password for password safe-keeping.