Categories
Personal Computing

Peabody is in the house

Dismal Wizard commissions Peabody, our new TrueNAS Core storage server. Peabody is a TrueNAS MiniX 5 slot system configured for file service and Roon Core service. This posts contrasts Peabody’s comissioning with that of our home brew server.

Photo courtesy of TrueNAS

As you recall, Sherman needed two disks transplanted. This work is complete and we have a third disk on hand in anticipation of the third going colicky. The long term plan is to add a second server and use Sherman as a backup machine via TrueNAS replication. The new server has arrived and is updating software. Read on for the rest of the story.

Revision History

  1. 2022-02-14: Original

References

  1. https://www.truenas.com/docs/core/gettingstarted/loggingin/
  2. iX Systems, TrueNAS Mini 3.0 Family Basic Setup Guide, Version 3.3
  3. Apple Time_Machine_SMB_Spec
  4. Jason Snell, Take Control of Photos 2.2, The Incomparable Inc, 2021.
  5. http://www.fatcatsoftware.com/powerphotos
  6. Kirk McElhearn, Take Control of MacOS Media Apps, Eyes of the World Limited, 2021.

The Take Control titles are available from https://www.takecontrolbooks.com/

This web shop handles ordering and fulfillment for the several authors and likely assists with preparation for publication. See the credits in the individual books.

iX Systems is a SuperMicro Fan

iX Systems is partial to Supermicro server boards and enclosures. They build most of their systems from private labeled SuperMicro OEM cases, chassis, and system boards. SuperMicro has a solid reputation in the server community.

So this means the BIOS screens and procedures are the SuperMicro procedures and IPMI is SuperMicro’s implementation of the Intel spec.

iX Systems Mini advantages

If you build home brew, you’re on your own to label disks and track which slot has which serial number. TrueNAS UI recognizes TrueNAS enclosures and shows which disk position has which serial number. This saves a lot of sticky tape or exploration when time comes for disk replacement.

Drive Bay area showing /dev/ad0 information

Assembly

Home Brew Again

I visited PC Parts Picker several times in attempts to spec out an AMD Ryzen Zen 3 system at small scale for disk backup. Most AMD motherboards are gaming oriented as mostly gamers home brew computers these days. PC Parts Picker serves this crowd primarily. Corporate users find it cost-effective to let an OEM assemble their machines and provide warranty and parts.

It was particularly hard to verify that a Ryzen motherboard supported ECC, a necessity for a system operating continuously. When we found one, usually by visiting manufacturer’s website and wading through specs, it was invariably not in stock. To support ECC, the traces have to be there, the machine check raised, the machine check forwarded, and the exception fielded by the OS and firmware.

I did look at System 76 Thelio systems. The AMD versions support ECC and some even come in large cases with enough storage slots to make a nice TrueNAS server. But they are also designed for movie editing, movie sound track editing, and other compute intensive and graphics intensive tasks. So they were pricey and needed storage added.

In the end, it proved too hard and I couldn’t beat iX systems pricing. So I ordered in.

Ordering In

iX Systems operates an Amazon web store that offers several Mini diskless configurations to which you can add disk drives. These systems are useful if you have disks on hand or are replacing a colicky system while retaining the pool volumes.

I ordered my system directly from iX Systems website for 2 reasons: I needed disks and I didn’t trust Amazon to pack and ship them properly. This turned out to be a wise decision as iX Systems disk prices were competitive and the Mini packaging is designed with protective flats in which they pack the disk carriers. When disks are part of the order iX systems mounts the disks in the carriers.

Early Fulfillment

iX Systems promised delivery 5 weeks after receipt of order and beat that by a good two weeks. Shipping was by FedEx home so delivery happened on Sunday. I let the system sit to warm up before opening packages and putting things together.

Adding the Drives

Do look at Reference 2 if this is your first time putting sleds in a chassis. The setup guide shows how to open the sled latch, slide it in, and close the latch. The lever must be operated to push the sled home and latch it in place. Once seated, a latch will click to keep the sled lever locked in the engaged position.

Initial Startup

Once the disks are in, the array can be installed in its shelving, Ethernet connected, power connected, and started. The machine will start, load the web console, and bring up the login screen. The as-shipped root password is in the getting started guide. Log in and wait for the dashboard to come up. On initial login and all logins, the dashboard checks for updates and will announce update availability in the system status tile of the dashboard. Stop and apply the update as iX Systems could have had the boot media in storage for some time.

Really Getting Started

The console sidebar is designed to facilitate installation by ordering topics top to bottom in the order in which you need to visit them and review or complete them. Mostly.

The new baby’s name is?

Once the updates have finished, you can visit the Network tab and set the host name and host domain. Until you do so, the host name will be truenas and the domain will be local. The system will advertise itself as truenas.local using Bonjour/Avahi/MDNS until you change these settings and restart.

The new baby’s address is?

Remember to configure your DHCP to assign a fixed host address to the new server as TrueNAS replication dialogs will not make calls to the resolver in a way that looks up a host name in MDNS. I set a fixed address in Ubiquity UniFi Network app by tracking down the machine’s port and setting an IPv4 address at the port. This makes an entry in the DHCP database. When the lease is issued or renews, the server will pick up the assigned address. This happened over night.

Replication does not use MDNS to look up host IP addresses so replication targets need a fixed address.

Users and Groups

I added users matching each MacOS account user but not the daemon accounts. TrueNAS will create the first user with number 1000, second is 1001, etc. TrueNAS will also create a matching primary group for each user.

Time Machine User?

TrueNAS has automatic configuration for Time Machine spares. No need to add users or groups for Time Machine as this will happen when the Time Machine shares are configured. Time Machine will run as the MacOS admin user (first one created at initial startup).

System -> General

There is is some work to do here but nothing unusual.

  • Enable HTTP to HTTPS redirect for the WebUI
  • Set the OS time zone. The system is born in LA
  • Set the system language and system keyboard map. They are initially selected as US English. If you use an Apple Keyboard, change the keyboard type to one having Apple keys.

System -> NTP Servers

Set the NTP server to the one of your choice. Pick one you know has a radio clock. Apple’s time.apple.com is a good choice as are the NIST public time servers and the US Navy’s public time servers. Consistent time values are important when using a password manager to compute 2FA number tokens.

Network -> Global Configuration

  • Set the system name (just the host name part).
  • Set the domain to local (as shipped value also).
  • Disable WS-Discovery announcements if there are no Windows systems present.

Network -> IPMI

Set the web interface IPMI password. Maximum length is 20 characters.

IPMI happens on the IPMI Ethernet interface which is not normally connected here at home. It is only needed for server recovery operations.

Storage Configuration

Reference [1] Storage Configuration section shows how to organize physical volumes into virtual volumes and assign them to a pool. Basically, put all the disks in then visit the Storage section to add a pool. This dialog has two paths forward, import the disks of an existing pool and creation of a new pool from virgin disks.

The work flow here is to create virtual devices and add them to a pool. A virtual device has one or more physical devices. A pool has one or more virtual devices. There are some configuration guidelines that should be observed. TrueNAS UI includes a “suggest layout” capability that knows the rules and can make recommendations.

This expert system will inventory the unassigned disks, suggest a configuration, and allow you to assign disks to it from the unassigned pool.

Peabody’s Physical Storage

Peabody has 5 WD Red Pro 6 TB disks installed. The pool configurator recommended putting these drives in a RAIDZ2 pool

Peabody also has 2 small SSD devices that are intended for the read cache and the write transaction log. Add pools for these devices and put an SSD in each. I picked the storage with this in mind.

Pool Creation

Select Storage -> Pools and enable create pool and click the Add button. This will bring up a form that adds disks to a volume and pool. There is an option to suggest layout. Give it a go.

Cache and Log Setup for the Pool

TrueNAS allows you to set up pools to cache the ZFS read metadata and ZFS write log transaction records. Like other ZFS resources, these are datasets in pools. The main pool creation form lets you create the cache and log volumes and creates datasets for use with the pool. Just designate the disks to be used. The 2.5 inch SSDs included in the configuration are used for this function.

Add the cache and log pools and add one of the SSDs to each. Accept the caution that these are solo device pools. That’s fine since the read cache is volatile and the log content is transferred to the main data pool making it ephemeral also.

For pools we now have just Pool1.

Dataset Creation

Each pool contains one or more datasets. Here, I use 2 types of data sets, a FileSystem and a ZVol. A ZVol is a dataset having a specified maximum size and reserved storage. ZVols are most commonly used for virtual machines and for Time Machine spool volumes. Virtual Machines is a dataset. Each VM has a ZVol in this dataset. VM creation creates the ZVol.

Pool1
...Clients
......FruitMachine
.........FruitMachine_Users
...VirtualMachines
......Roon_VM
...FM_TM

Data Pool Encryption

I encrypt the data pool, not because I’m paranoid but because it makes disposal of the disks convenient when they go bust. Note when, not if. There are only two kinds of disk in the world, those that have failed and those that are about to fail.

Do download and save your pool encryption keys. I keep mine in 1Password.

By AES-256 encrypting the pool content, the data is protected should somebody go dumpster diving or your disks end up in the local suk.

Moving Time Machine

Time Machine decided to be fussy. It didn’t like new storage on Peabody. Time Machine would not use a newly created dataset and a newly created share. When creating the share, be careful to open the advanced options and enable the Multi-user Time Machine check boxes there. These set a number of capabilities that Time Machine requires.

It won’t play

I created the new share as recommended in the TrueNAS core user guide. Apple stuck its tongue out at it. There was no diagnostic information in either the message presented by Time Machine or in the console log.

First Workaround, no joy

  • Disable the shares and restart Peabody.
  • Create a one-shot replication task to move Sherman’s working Time Machine dataset to Peabody.
  • Let the dust settle. This will take a while as it is over Ethernet and encryption is happening.

When the dust settled, the replication failed.

Second Workaround, Joy

Consulting with TrueNAS community, they told me to do what I’d thought done. So I tried again verifying the things mentioned in the coaching. I created a new dataset that was a great grand child of the pool. Same symptoms. Time Machine didn’t like the share.

So I created a second dataset that was a child of the pool and created an associated share. This time joy. Checking Sherman, the working dataset on Sherman was also a child of the pool.

Don’t double up encryption

If the dataset receiving Time Machine backups is encrypted, there is no need for Time Machine to encrypt its archives. One less key to manage. In general, minimizing complexity maximizes success in network land.

SMB Sharing for Apple Time Machine

TrueNAS 12 handles share creation for Time Machine nicely. Just select purpose Multi-user Time Machine at initial creation. This action sets the required advanced options. Don’t forget a description. Then save the new share.

Photos

In Reference [4] Jason Snell advises that photo libraries should remain on the internal disk because Apple has reworked Photos App to take advantage of APFS snapshots and copy on write capabilities. Although TrueNAS OpenZFS has these capabilities, they are not exposed over an SMB connection.

Dismal Manor practice is to keep photo libraries by decades for the migration to Peabody. This makes the files easier to search and easier to use.

Working with libraries

Jason recommended Power Photos [5] to work with photo libraries. Jason covers multiple libraries, splitting libraries, combining libraries, etc in [4]. It turns out that Power Photos is also good at searching out duplicates in libraries and makes removal of the redundant copies easy. The Power Photos duplicate removal library is much more convenient than that in Gemini 2.

Where they ended up

Photo libraries are on the Mac in ~/Pictures/Photos as Apple expects. I also marshaled the older videos into a single library of their own. Also everything I dragged off of a camera with Image Capture is in an Image Capture Photos library. All on the Mac.

Music Media

I have music media from several sources. I have purchased a small amount of media from Apple Music, typically an artist like Kim Churchill whose label distributes product in his home country (Austrailia) but not internationally. Most music in the library was purchased on CD media and imported or purchased from Qobuz.

Disk Commander view of music organization

CD Media

I have a large amount of music (600 CDs or so) that I’ve purchased over the years and transferred to Apple Music to play. For the last 10 years or so, I’ve streamed the music from the Mac using one or another gadgets (Logitech Squeezebox stuff originally) and for the last 5 or so years Roon Labs Roon.

Almost everything in the Apple Music Media is from physical CD that I have transferred. I keep this music duplicated in the primary server to isolate it from acts of Apple that may or may not occur. They can always change their minds about how Music works or behaves.

Apple Music Media Purchases

So I’m careful to keep my purchased music separate from my iTunes store purchases as Apple will let me pull them down again or play the Lossless version in Music. Music works well with Roon and AirPlay capable Roon endpoints.

Qobuz purchases

The Qobuz App will let you determine the albums and tracks that you have purchased but transfer of an album from the shop is possible using only the time-limited link in the fulfillment E-mail. Loose a Qobuz purchase and it is gone. So I keep the Qobuz purchases separate so I know what they are.

Roon Focus on Source

Roon lets you search your library by source. You can pick encoding type (FLAC) or purchase source (Qobuz) or streaming source (Qobuz). Roon lets you export to a CSV. All of this data is produced from the Roon database. It is important to back up the database (to migrate a server) or to rebuild your music library following a mishap at home (hurricane, wild fire, house fire, etc).

Roon Album Focus capture showing selection criteria

If Roon is playing a local copy, it will show the path of the media being played. Media paths can be included in the save.

So, I keep my Qobuz and CD stuff separated so I can replace the Qobuz stuff should the need arise.

ZFS Backups

TrueNAS uses OpenZFS snapshot replication for backing up user data to another ZFS host. The configuration process pairs the snapshot task to the replication task. Completion of the snapshot triggers the replication.

Sherman’s new life role is as a backup host for Peabody so Sherman will receive replications in its large pool. There is sufficient uncommitted storage for this purpose. Sherman will continue to snapshot and replicate to the internal backups pool.

Configuring Sherman

First, I created a new dataset Peabody in the FreeNAS pool. Then, I adjusted Sherman’s snapshot task to exclude Peabody from the recursive snapshot set. Sherman continues to back up to the Internal Backups pool.

Configuring Peabody

On the upstream end, I configured a new snapshot task on Peabody that would snapshot the Pool1 pool in its entirety. I added a replication task to transfer that snapshot to the newly created FreeNAS/Peabody pool on Sherman.

Cleanup of Sherman and Fruit Machine

Eventually, I’ll remove Sherman’s unused virtual machines and the associated datasets and legacy user data once I’m confident that everything is accounted for. As part of this cruft roundup,

I’ll also clean pictures and documents on Fruit Machine. I have 20 years accumulation in these locations. In some cases, data may not be recoverable because the applications date back to PowerPC days and are no longer available.

Historical Notes

Fruit Machine was named while reading British submarine novels. British Submarine slang refers to the torpedo fire control system, particularly the target motion analyzer portion, as the Fruit Machine. I needed a name for the new machine, my first with an Apple Silicon M1 processor and GPUs so the name seemed rather obvious. Later I learned that fruit machine was also a bit of unpleasant LGBTQ slang. Hopefully we can keep our overloaded slang sorted in context.

The names for Peabody and Sherman come from the old Rocky and Bullwinkle and Friends children’s TV series which featured cold war satire disguised as children’s cartoon characters and story lines.

Mr. Peabody was a dog who had a time machine and a star pupil boy Sherman who took rides in the time machine (named the Wayback Machine) to experience history first hand. Peabody and Sherman comprise the Dismal Manor Wayback machine. This a somewhat high falluting title for a glorified digital shoe box distributed across the 3 machines.

By davehamby

A modern Merlin, hell bent for glory, he shot the works and nothing worked.