Are you satisfied with your backup?

Back in the day, Dismal Wizard administered VAX/VMS systems for several projects. Every time you used VMSINSTALL it would ask “Are you satisfied with your backup?” I can’t recall ever saying no. This was back in the day of CDC Storage Module Drives, backup to open reel tape, and more recently to TK-50 cartridges on the small systems.

This article reviews some important concepts. We make backups to protect ourselves from gremlins. The gremlin may be an inadvertent deletion, accidentally replacing the filesystem on an active device, hardware failure of the disk, a failing processor writing buckshot to a disk, or theft, fire, and flooding resulting in complete lose of a volume.

Another peril we face is that the backup media itself is corrupt. In one horror story, we discovered one of our minicomputers had two controllers strapped to the same device address, the system disk, and the tape drive we used to back it up. We ran the backup and bam, an OS crash. We found backup records had been transferred to the disk wherever the heads happened to be. ++UNGOOD. And of course, some disk IO was going to the tape.

So we use redundancy and diversity to ward off misfortune when misfortune and mishap happen. The spatial diversity of off-site storage protects from data loss when the facility is damaged. Retention of multiple backups protects when mishap is discovered sometime after it has happened. And diverse software and local media protects from bug related mishaps.

In this post I introduce Arq Backup, a new addition to the Dismal Manor mishap and misfortune recovery plan.

Backup Diversity

One is none and two is one …

Back in the old days, we backed up. to magnetic tape. Before about 1995, that meant handling tape, taking the machine down to single user, writing a save set to tape, labeling and shelving the tape,etc. It was not uncommon to miss backups, have the tape go flaky on the shelf, forget to change the label, miss-shelve the tape, or if part. of rotating pool, to just get old and worn/dirty and fail to read. With tape, we needed multiple copies as, more often than not, the most recent tape would fail to restore. Or we couldn’t find it.

Diversity applies to storage media and to backup tools both. That wise old technician Murphy wrote,

If anything can go wrong it will.
If something can’t go wrong, it will anyway.
When things appear to be going better, you’ve overlooked something.

His complete dissertation appears in [1]. Some believe that Murphy was an optimist.

Although much more reliable than mini-computer tape backups, modern backups to disk still go wrong.

So if TimeMachine, fine product that it is, does all local backups, do we really have a local backup? Of course, there are no bugs in TimeMachine.

If BackBlaze, popular and well priced service that it is, does the off-site backups, do we really have an off-site backup? Students of Dr. Murphy know the answer, “If anything can go wrong, …” It’s all up there. But can we get it back? Will our ISP cooperate or yell at us for busting data caps?

So, how do we achieve diversity.

Use multiple media types and locations
Use multiple backup tools

Originally, we were using just Time Machine and making a single local backup to an external disk. Did we have a backup? One is none. In spite of that fear, Time Machine saved my bacon once when the internal disk in my old G4 Tower went tango-uniform (look it up, this is a family blog).

And I’ve done system migrations twice using Time Machine. I’ve been lucky. I don’t like to rely on luck. Lady Luck has a way of going missing when needed.

Our Backup Oddessy

In the beginning I built Bacula and tried to backup to a Linux box. This sort of worked. Then Apple added Time Machine to MacOS in 2007 or so. I bought a Drobo II and put 2 disks in it and let ‘er rip. I bought a second Drobo for media (iTunes and Photos). After 5 years or so, disks began to fail. I replaced them, and Drobo did its magic.

On-Site Storage

When I retired, the Drobo II boxes I was using for media and backup storage had become obsolescent.

They were small,
they relied on proprietary stuff, and
the FireWire 800 interface was starting to be phased out.
And Drobo had put the II in end of life as the hardware architecture was old, users were retiring them, and it was difficult to pare down the later software to run on the old iron.

Although they worked reasonably well, they were a bit of a white elephant. They relied on a proprietary driver and a proprietary control panel that was starting to get fragile. MacOS evolution was threatening to break both. Oh, and FireWire was endangered by change. So I soldiered on via USB for a bit until Drobo stopped updating the kernel driver and control panel.

So I needed to replace the Drobos. I came up with the following requirements.

Robust like Drobo but not proprietary.
NAS as opposed to DAS as interfaces and management were standard at the client level
Purpose built OS distribution, not Linux and CIFS
Easy expansion and management like Drobo but without proprietary add-ins

I settled on TrueNAS after watching a lot of YouTube video. I built my first TrueNAS box from new bits following DIY NAS building guides [2]. During the pandemic, I looked to build. a second system to serve as primary with my original becoming a replication target. I bought from iX Systems as the PC new parts market was chaotic at the time. From experience, I knew I couldn’t beat iX Systems price.

Off-Site Storage

While I was doing the research to build the TrueNAS box, I came across TrueNAS Forums posts mentioning BackBlaze for off-site backups and some intrepid folk (admin pros?) were backing up their TrueNAS servers to BackBlaze B2 cloud storage. BackBlaze is quite good at this. Current TrueNAS has stuff to replicate to B2 but it is expensive as BackBlaze charges for directory operations in addition to data I/O and storage. But they had a $60/year plan to backup a single PC to Apple specifications (skipping the things Apple says to skip). So I had a go and tortured the hell out of Cox. I’ve not needed to restore from BackBlaze so I have no idea if it is really there but I can look at it on the WWW using their browsing interface (pretend to do a selective store).

BackBlaze Restore Model

BackBlaze does an interesting thing. To restore, they’ll copy your save sets to Apple formatted USB media, ship it to you (for a charge). You do the restore, return the media, and receive a credit for the cost of the media. This works pretty well. Today, a 2 TB LaCie 2.5 inch disk in enclosure is $169 or so. You restore by copy. For a small selective restore, the BackBlaze UI will do that across the Internet.

Do I really have a backup?

Two is one and one is none. All my eggs were in the Time Machine basket locally and in the BackBlaze basket off-site. One is none. One of Marco Arment’s (OverCast developer and Accidental Tech Podcast) peeps mentioned Arq, its ability to save locally, its ability to save to various cloud storage providers, and that it worked. Not flashy but trustworthy. Particular Marco Arment is a celebrity endorser of Arq Backup. So I snagged a copy.

Arq Backup’s thing (besides working) is that they will do off-site backup for $60 per year per TB stored. Additional TB are $5/month by the GB ($0.0005 per GB). So I snagged a demon copy and had a go. Arq proved to be quick to set up and needed about 2 days to shovel everything to their storage location in South Carolina. I’ve not needed to do a restore but that looks easy.

I elected the Arq Backup provided on line storage option as I had no desire to become a cloud storage expert and it provides Arq’s developer with continuing income. Stefan Reitshamer is a solo independent developer relying on Arq Backup’s sales and online storage for income..

So, I have 2 Off-site Providers

I should have 2. Both rely on proprietary applications to talk to Mother. Not completely ideal but workable.

Now, I need a second On-Site backup. Marco’s peep mentioned Arq Backup again as a Time Machine replacement. So I reformatted the USB 8TB and let Arq Backup have it. Arq Backup appears to be chugging away. The disk is slowly filling. In the time to write this, 200 GB are over there.

Both Time Machine and Arq rely on APFS snapshots to do their thing. Hourly each takes a snapshot and transfers it to disk in a replication sort of way. The underlying OS copy on write and journal are doing most of the hard work of determining what to transfer.

Apple saves each snapshot to the Time Machine volume. The Time Machine UI is Finder-Like. It shows you the file system as it existed at the point where the snapshot was made. You determine what to restore. You can restore earlier versions. NTFS is also copy on write and supports snapshots but Microsoft has its own trademark for them.

So where are we?

So, I’ll try this until the new year. I was thinking Arq would replace Backblaze but, one is none. It appears affordable to keep 2 off-site backups and 2 on-site backups, one on USB storage and One on Peabody and Sherman.

Have I tried a system restore?

No, I’ve not. I should buy some storage and have a go at it. I’m pretty confident in Time Machine as it has served well 3 times in the past. Arq and Backblaze have not been tested.

One limitation of both Arq and BackBlaze is that they don’t restore the system stuff. Apple MacOS system recovery procedures are used to restore the operating system. You then restore your Time Machine image from within system recovery.

Arq and BackBlaze need to be installed, accounts revived, and their recovery procedures followed to restore your user data.

Why TrueNAS

TrueNAS is IX Systems product version of FreeBSD with the ZFS filesystem. ZFS is a copy on write file system combing logical volumes, error correction and recovery, and replication. A FreeBSD ZFS pool can be configured replication to another pool in the same machine or to another machine reachable by network. Replication can be to another machine on the LAN or to an off-site machine.

IX Systems provides a web UI for system management that makes management of FreeBSD and ZFS accessible to old modeling and simulation retirees like me. I have 2 TrueNAS servers, a primary and a secondary that serves as a backup of the primary. The primary is also my NextCloud server. The secondary is my Channels DVR. There is a robust user community and IX Systems offers Core for DIY use and Scale and Enterprise for supported corporate and enterprise use. An active volunteer community of sys admins fields questions to support Core. The Core deployment and usage guide can be used for most management tasks.

TrueNAS Sharing has a settings profile for an SMB share that configures it to meet the needs of Time Machine. One click and you’re good to go.

Your Mac is a Family Affair

The administrative user restores MacOS and his user data. Time Machine always functions in a user context. First user Time Machine save sets include the system administrative data. Subsequent Time Machine save sets include only the user’s home directory tree. Arq and Backblaze are likely the same. If you are sharing a Mac with the family with multiple user logins in use, you need to tread with caution. Joe Kissel’s book [7] and Securing Your Mac [8] are essential reading to set up local user accounts, family sharing, and user backups.

Arq Backup Lessons Learned

Now that I have some accumulated backup history, I looked to see what had been saved. Here’s what I found.

Arq Backup Tries to Save Everything

By default, Arq tries to save everything. Arq will try to save its own output volume. So you have to exclude it. Or it will chase its tail. This is true when saving to cloud storage, NAS storage, or to local external storage. So check your exclusion list when creating a new storage plan.

*Arq Backup Status and Restore View*
*Backup to Arq cloud storage*

Arq Backup Skips Some Things

On MacOS, Arq Backup has about 69 categories of stuff it skips specified by file system wild cards. TimeMachine skips more stuff and it is OK to skip that stuff. It’s mostly log files, cache files, and such. There’s a check box for that. The figure shows how to check.

*Arq Backup exclusion settings open to edit*
*Note the check box enabling Time Machine skipping.*

Arq Selective Restore

Arq Backup restore sessions can be configured to restore to a selected point in time for which there is a save set. You can restore the entire save set from its root or any branch of it, or just a leaf or two.

*Arq Backup External Storage Save Set*
*/Users directory view*

The restoration view of an Arq Backup Save Set is a complete snapshot of the file system at the time indicated on the left pane. The right pane behaves like Finder. Open directories until you find what you need.

And, Finally, a Test Restore

Now that Arq Backup had settled, I decided to try a partial restore of one of the local save sets. You browse just like in Time Machine but without the heavy-weight interface with its Star Wars view of the snapshots. Select a snapshot and browse through it. Select the branch you wish to restore. Arq Backup asks where it should go and gives some common sense choices including

Desktop
Network storage
External storage
System volume

Select the volume, then directory, and restore. For my test restore, I reformatted a media disk as APFS and saved a photo to the root.

Revisions

References

You’re New to Macs