Computer Backup at Dismal Manor

There are only two kinds of computer storage, those that have failed and those that will fail. Remember Murphy’s Laws.

1. If anything can go wrong, it will.
2. If something just can’t go wrong, it will anyway.
3. When things appear to be going better, you’ve overlooked something.

In this post I’ll describe Dismal Manor disk backup scheme.

Revisions

1. 2026-08-06 Original

References

1. https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
2. https://www.illumos.org/books/zfs-admin/gavwg.html
3. How to Reinstall MacOS
4. Using MacOS Migration Assistant Mac to Mac
5. Working with APFS volumes in Disk Utility
6. https://bombich.com/doc-pdf/ccc3-documentation-all.pdf
7. https://www.shirt-pocket.com/guide.php
8. Apple Time Machine User Guide

Why back up

We do disk backups to protect our information from mishaps. A mishap can be unintended deletion or overwriting of a file, failure of a disk or other storage, or corruption of data by a not quite broken processor, main memory, or disk data path.

Back in the day of 80 megabyte storage module drives, I was software integrator for the modeling software of an operator training simulator for a nuclear power plant that my company was developing for one of our reactor owners. I had stayed late to do a build that finished at around midnight. Being tired, I said “F-it, nothing will happen until I come in tomorrow”. The head crash happened as I was driving to work.

Some time later, I had finished a build and was doing a backup of the build pack. The hardware bubbas had been working on this machine. They had managed to strap two channel controllers, the one with the disk and the one with the tape to the same channel address. Eventually, a write to tape went astray and ended up on top of the disk super-block and directory.

And on another occasion, backup could not read the tape. Fortunately, the OS copy utility could. Each file had a metadata record followed by the data as blocked disk sector images. I was able to figure out how to use OS copy in a way that would recover the data on the tape.

And Combustions’s CDC 7600 floating point divide unit became colicky. It would occasionally mess up a division without throwing a machine check. The operators and admins realized something was amiss when SCOPE couldn’t find one of its system files. The OS used the floating point unit to do 48 bit integer divisions in the file hashing algorithm. A bad divide was a wrong hash that could not be found in the disk directory. The recovery was pretty simple once the floating point divide unit had been fixed. The hard part was figuring out which production runs were NFG as a result of division troubles.

The 3-2-1 Strategy

In Reference [1] , Backblaze outlines the 3-2-1 backup strategy first postulated in the early Naughts. The argument is to have at least 3 backups on two types of media (or 2 devices), and 1 offsite copy.

The thinking was one copy offsite in case disaster befell the site.

Use two types of media or two devices for diversity of storage. For example, one copy on an external disk and one copy on a NAS.

Have a minimum of 3 copies.

3-2-1 at Dismal Manor

Dismal Manor has 4 computers, Fruit Machine the Apple M1 Silicon iMac I’m using to write this, an AMD Zen 4 machine that is Rocky, and 2 TrueNAS Mini storage systems (SuperMicro small servers). The Apple machine uses Apple File System. The three Linux machines use OpenZFS for 4 drive storage arrays. Each storage array is a ZFS pool holding multiple datasets. All four filesystems support snapshots.

Dismal Manor uses several backup programs

• Apple Time Machine
• Backblaze Personal
• Carbon Copy Cloner
• SuperDuper
• Linux TimeShift, a TimeMachine work-alike.

The TrueNAS machines are not backed up at the moment as they are used as backup media. Linux TimeShift backs up Rocky to an external SSD.

The Dismal Manor Way

• Fruit Machine makes Time Machine backups to an external disk and to shares on each of the TrueNAS servers.
• Carbon Copy Cloner and SuperDuper! also make backups to shares on each TrueNAS server.
• This do it twice scheme proved simpler than setting up replication from the primary TrueNAS system to the secondary using ZFS send and ZFS receive jobs. Doing it twice means that the second ZFS backup is independent of the first.
• Backblaze makes an off-site backup. When recovery is necessary, Backblaze will express a disk containing your latest image.

Backup vs Recovery

Dave Nanian of SuperDuper! fame differentiates between backups and recovery volumes. Backups are used to recover from minor mishaps such as deletion or overwriting of a file or a directory. Recovery volumes are bootable external disks that hold a recent system image and file system image. Time Machine or Linux Time Shift make backups that allow you to retrieve the current version or earlier versions of a file or directory. Carbon Copy Cloner and SuperDuper! make a complete copy of the file system and optionally make bootable media.

Recovery

The basic MacOS recovery workflow is as follows. Back in the day of tower cases, disks were user replaceable. Today, the system storage is soldered on the system main board. Apple Genius Bar makes a main board transplant for you to replace the failed disk. The new main board will include the MacOS recovery system.

To enter recovery, press and hold the power button until the “Loading startup options” prompt appears. Release the power button. Choose the system recovery option from those presented.

1. Replace the failed disk.
2. Restart the Mac from the Recovery Partition.
3. Follow the prompts to reinstall MacOS from the network.
4. Use Migration Assistant to restore user data from Time Machine or a duplicate user data image.

Why ZFS

OpenZFS is the most robust file system available for personal computers and servers. ZFS is a journaled file system. All transactions are written to the journal first. Periodically, the journal transactions are committed to the file system proper. Like APFS, it is copy on write and supports snapshots.

Filesystems and ZVols

ZFS has the ability to combine multiple physical devices into logical devices. Logical devices are grouped to make a storage pool. A pool can contain one or more file systems. The file systems may be thin provisioned (datasets) where storage blocks are added as needed or preallocated (Zvol). Zvols are used where a hard upper bound on size is desired. File Systems are used when the ultimate size is unknown at creation. Zvols are frequently used to keep TimeMachine from eating a pool.

Redundancy

When a pool is created, it can be created as RaidZ1, RaidZ2, or RaidZ3 indicating the amount of redundancy in the dataset. The number indicates the number of redundant copies available in the Zpool. This is an indication of the number of disk failures that the pool can tolerate before data is lost. RaidZ2 is a nice balance between redundancy and capacity for small pools (4 disks).

ZFS administration commands allow you to replace a disk and reconstruct the data that was on it from the good copies still present. Reconstruction is called “reslivering the pool”. The resilver operation is disk read-write intensive and takes a while. It is not unknown for a second drive to fail during the resilver operation (this happened here once).  For this reason, most experienced admins recommend RaidZ2 as a minimum level of redundancy.

Data Integrity

OpenZFS includes both metadata and data block checksums. This permits the filesystem to verify the data integrity of each file’s directory entry and each of its data blocks. This operation is called a “scrub”. I setup my systems to do scrubs weekly.

ZFS is copy on write. When a block is changed, a new copy is written to disk. The old copy is retained. Together, copy on write and snapshots provide robust protection against unintended or malicious alteration of data. You can always revert to a snapshot when the problem is corrected.

Notifications

Stuff happens. When it does, OpenZFS can send you notifications, normally by E-mail but can also use the host operating system’s notifications system. All notifications are also logged. Admin commands may be checked using ZFS terminal commands.

Snapshots and Replication

OpenZFS, like APFS, supports snapshots at the Zvol and dataset level. Think of these as checkpoints. The file system can be reverted to any snapshot.  Once created, a snapshot is immutable. 

The ZFS send and receive commands allow you to transfer a ZFS snapshot from one instance of a filesystem or volume to another. Together, send and receive are the primary mechanism for backing up ZFS datasets (filesystems or Zvols). A replication may be performed internally to the system, between local systems, or between networked systems. Replication is a handy technique for maintaining off-site backups. Leave a machine with family or friend and replicate to it periodically. 

Cockpit and 45Drives Houston

45 Drives, the “Big, Strong, Fast” people, have developed Cockpit modules for OpenZFS. They allow you to create volumes, add them to a pool, and create filesystems and zvols in the pool. You can also replace failed devices.

Some Limitations

All devices in a pool are the same size. They may not actually be but ZFS will use only of the space of the smallest device on each physical disk that is larger than the smallest disk in the Z-filesystem.

Backup Programs

We use 4, Apple Time Machine, Backblaze Personal, Mike Bombich’s Carbon Copy Cloner, and Shirt Pocket Software Super Duper. 

Backblaze Personal

Backblaze Personal does network backup to Backblaze servers. We use the versioned method allowing us to restore to checkpoints determined by Backblaze. Backblaze Personal is currently $100/year plus a paid option for unlimited snapshot storage. This second feature lets you revert to a prior version should the most recent and some of its predecessors be corrupted by hardware trouble or exploit.

Carbon Copy Cloner

Carbon Copy Cloner can back up to local media, local shares, and cloud storage services that are not iCloud. The important thing about Carbon Copy Cloner is that it builds a duplicate of the protected volume at the destination. The directory structure, times, owner, group, and other permissions, and access control lists are preserved. There’s just a tree, no tarfile, no dumpfile, no secret archive, just a directory tree that makes sense to Migration Assistant and to the destination file system, mostly.

I use CCC to backup to a local USB disk and to a SMB filesystem on Rocky, our 45Drives home lab machine. 

Carbon Copy Cloner can construct a bootable MacOS volume and will transfer the entire file structure of the source disk to it. Carbon Copy Cloner has been updated to work properly with APFS disks as source and destination volumes.

SuperDuper

SuperDuper! was just updated to Version 4, a complete rewrite to make the product more APFS friendly and more Apple Silicon Mac friendly. Super Duper replicates the source volume’s structure and data on the destination volume. Again, owner, group, permissions, and access control lists are preserved. And the destination replica makes sense to Migration Assistant.

Time Machine

I use Time Machine to backup to a local USB disk and to SMB filesystems on Peabody and Sherman, the two TrueNAS boxen.

Time Machine maintains a series of snapshots in its save set. When Time Machine runs (usually hourly), it begins a new APFS snapshot on the system disk and saves the previous snapshot to the external disk.

The Time Machine user interface lets you restore any version of the file that is in the archive. It is good for oops recovery.

External USB Disk

The external USB disk is an 8TB LaCie external disk. It has proven robust over many years of service. The original disk is still in it.

Reference 6 gives recommended practices and procedures for formatting a disk as an APFS volume. The recommended practice is to format the disk as a single container with multiple APFS filesystems in it.

The caveat here is that APFS file systems are sparse-provisioned. That is, storage blocks are allocated as needed. Some care is needed to avoid over committing the device. Apple recommends that a Time Machine volume be a couple of times the size of the volume being saved. My 8 TB LaCie disk can hold one backup set for the 2 TB MacHD and one for the 1 TB Music and Photos Volume.

Recovery

Apple Silicon Macs have recovery firmware built in. The recovery software will do a network boot, install MacOS, and restore user data using a program called Migration Assistant.

Reference [3] gives Apple Support instructions for running MacOS Recovery. Reference [4] explains how to use Migration Assistant to restore user data.

Migration Assistant Data Sources

Migration Assistant can transfer data to a new MacOS installation from the following sources

• Your old Mac
• A Time Machine archive on an external disk or network share
• A disk image like those created by Carbon Copy Cloner and Super Duper.

Disk selection is simplest when the data is on an external disk

Stock replacement disks

I got into the habit of stocking replacement disks of the largest size currently in the Raid-Z pools. When one fails, I have a ready spare. Following TrueNAS or Houston UI procedures, I offline the failed volume for replacement, remove the bad disk and slip in the spare. Once the spare is up and identified, I add it to the pool using the UI replacement procedures and start the re-silvering operation. Re-silvering generates the information that was on the failed volume from the redundant information in the pool.

It is not uncommon for a second disk to fail while the replacement is being re-silvered. If the file system was RaidZ2, there is still one good set of data in the system. Let the first replacement complete rebuilding. Then replace the 2nd disk.

When the dust settles, order new market catch of the day disks for spares.