Earlier this week I updated Peabody and Sherman to TrueNAS 13. Following the update, I discovered a couple of issues. First, replication from Peabody to Sherman had stopped. That is, Peabldy was no longer transferring backups to Sherman. And Time Machine had stopped. What the hey.
With some community help, it turned out that I had made a mistake in doing the update and that I ran into a somehwhat hidden design feature.
The Mistake
Every time you open the UI, TrueNAS checks for updates. Each version of TrueNAS belongs to a “train”. Production versions come from the Release train. Preview releases come from a different train. (Preview?) About a week after YouTube folk had mentioned that TrueNAS 13 was released, I did the update. Not carefully reading the Release train update description, I switched from Release train to Preview train to pull and install the update.
When TrueNAS switches releases on an Update, the Release Channel update will be from release N to release N+1. The update availability notice will state that cleanly along with having revisions numers and other build provenance in both the from and the two parts. So read these carefully. You may want to hold back on a version migration.
I updated Sherman first since he could be rebuilt easily. Then I updated Peabody. Then the alert Emails started coming. When TrueNAS spazzes out, it sends you a notification via your configured Email transport provider, in our case Google. (There is no escape.)
The “Feature”
TrueNAS allows you to create a filesystem for use by Apple Time Machine as a backup store. When you create this store, you create it in the pool. Then you create a share for it. When you configure the SMB share, you specify that it is to be used for Apple Multiuser Time Machine. This sets some SMB options needed for Time Machine to operate properly.
Time Machine woke up, created the davey filesystem inside the FM_TM filesystem and went to work. Time Machine tagged davey with a one terabyte quota. Without telling anybody. Shortly before TrueNAS 13 had released, Time Machine hit quota and started failing. Not much in the error message. So I mounted the volume, got info, and saw zero free space!
The Recovery
Recovery turned out to be straight-forward but ate the afternoon.
Replication Failure Recovery
The fix here was to revert to the previous good boot environment and restart the machine. Check for updates. Update from the Release channel rather than the preview channel.
Oh and update both machines as they must be running the same release family (both 12 or both 13, not one 12 and one 13). WebUI did this easily.
Time Machine Full Recovery
The fix here was easy but took a bit longer to find as TrueNAS WebUI does not make it obvious that there is a user quota on a filesystem. The switchology is clear but the indications that it is present are low key. Check each container in the storage tree to see if quotas are set. In my case, it was the leaf node that had the quota. Quotas are by user, so open the quotas table, check the user (by number since several usernames could have the same number), and up the quota. I doubled it.
And On the Client
My record collection lives on Peabody. It is not backed up in Time Machine. (I need to fix this).
But I buy most music in digital form these days. A title comes into downloads and just sits there. I also tinker with Raspberry Pi a bit. Each new image lands in downloads. And sits there and sits there.
Well guess who left downloads in the span of things Time Machine was backing up. You got it.
So I went to Time Machine and told it to add Downloads to the exclusion list.
I also created a Backup Never directory that I added to the exclusion list. This gives me a place to put temporary user files that I don’t want to find their way into Time Machine. This is useful when creating encryption keys, etc that will go into BitWarden or 1Password. These shouldn’t be lying around in a relatively accessible working directory. A tip of the hat to the Accidental Tech Podcast people for this idea.
Dismal Manor Dispatches 