For the past several months, I’d had a full volume on my Gen 2 Drobo. What’s a Drobo? It is a Firewire/USB external disk drive designed for use by small offices and professionals. Drobo is popular with photographers, musicians, video editors and such as an external storage device because it has some interesting properties.

It is disk failure tolerant
It is easy to expand
Failure recovery is easy
You don’t have to be a professional storage admin to manage it

I’ve had this device since the Fall of 2009 when I purchased my current Intel Mac Mini. For the past several months, the system had been giving me fits with slow performance and unfinished disk backups. Being newly retired, I had the time to look into these problems in depth.

Some History

I’ve been a Mac OS X (say ten) user since 2002 when I purchased my dualie G4 mirror door machine. I’ve been migrating stuff forward through 8 OS X updates and a hardware swap so things had gotten crufty with abandoned executables, senseless start up items, etc, passe widgets, etc. Time for a house cleaning.

Second, in 2012, living in the east coast hurricane alley, I decided off site disk backup was good and began using Carbonite for this. In 2013, I became curious about Pogoplug and began using it too. Thus, I had 2 off-site backup daemons churning the file system.

In June 2013, the Time Machine volume became full in a way that caused it to mount Read Only. The drive reported write protected status to Time Machine rather than end of medium status confusing Time Machine’s end of medium logic so it just sat there and told syslog it was in trouble. No really clear user messages

Summer of 2013 also saw the Snowden Disclosures about NSA agressively trying to spy on all Internet traffic, FISA, Patriot 2, etc called to our attention. I decided to discontinue off-site backup to minimize exposure to acts of my government.

Winter Cleaning

I went through the start up items using Clean My Mac and disabled those not being used. Once comfortable that these changes are safe, I’ll use Clean My Mac to delete the old items. Clean My Mac knows what can be bushwacked and what must be kept around. For example, it won’t propose deleting or disabling the OS X UI server (a really bad thing).

I cleaned up old stuff from my TIvO days, iSTAT menus days, etc. Anything not essential to normal operation that I was able to identify as a 3 rd party add-in became history. Carbonite gone, PogoPlug gone, Quicksilver gone, ISTAT menus gone, SMART monitor gone. Amazingly, with the the third party crud gone and the external disks having free space again, life was good.

How Time Machine works

Time Machine is a really cool Apple OS X system service that has been around for several years. Time Machine is designed to back up a UNIX file system to external USB/Firewire disk storage. It doesn’t do tape because appropriate tape drives are $5000 devices that require SCSI interfaces not found on personal computers. The way TIme Machine works is to maintain a current virtual image of the file system on the external drive by making a baseline backup plus hourly changes. As the program runs, it consolidates the hourlys into daily snapshots, weekly snapshots, and monthly snapshots.

The way it does this magic is to make a copy of each unique version of a file to disk hourly. Files that don’t change are written just once. Files you are working on are written hourly. Say you are writing a letter and do the following.

Save the template
Edit the template to make something useful
Save the draft
Print the draft to preview for proof reading
Edit the draft
Save the draft
Print for mailing

Depending on how this activity spreads out over time, Time Machine will capture one or more versions of the letter, the final draft which is persistent and perhaps one of the two working copies.

How does Time Machine keep things straight? Behind the scenes, Time Machine keeps a version history for each file listing the current version and each available previous version. This history is organized for easy query by directory and date. When you enter Time Machine, it opens the directory and shows you your home directory for the current day in a Finder style window. You can poke around the current time in Finder. To move back in time, you use a slider at the right side to pick the available date and time.

Time Machine uses hard links to construct these virtual views in the Time Machine universe. This allows normal UNIX file operations to show what is available and to restore a directory or file becomes a simple UNIX copy.

More about the full volume

Drobo originally held the following disks

1TB Western Digital Caviar Green
1TB Western Digital Caviar Green
2TB Western Digital Caviar Green

Drobo proposed creating a 2 TB volume but reports 16TB total size, the maximum for this model. Drobo takes some off the top to store metadata so 2 TB is actually 1.8TB or so. Thus, to have a true 2 TB HFS+ volume requires more than 2 TB of installed disk. With the drives I had, the best Drobo could do was 1.8 TB and if I added a disk, it would give me a true 2 TB volume. The bad bit was that it went write protected rather than staying read write and returning end of medium status.

Drobo is file system aware unlike RAID which is disk block aware. Drobo tires to be smart about how it splits up file blocks across the available disks and how it creates and organizes forward error correction data for the file data. When there are two identical disks as the volume started out, Drobo mirrors. When I added the 2 TB disk, it reorganized the files and forward error correction to be spread across all 3 disks in such a way that the volume could be supported by any 2 of the disks.

Because this volume was made from 1 TB, 1 TB, and 2 TB, the Drobo tax still prevented creation of a 2 TB OS volume. About 1.8 TB were available because Drobo needed 200 GB (about 10 percent) for its internal record keeping.

Adding the third disk

In December, I ordered a 3 TB Western Digital Caviar Green disk from MacSales.com. When it came, I allowed it to warm up to room temperature and installed it. Drobo recognized the disk, filled out the 2 TB HFS+ volume and proposed creating a second smaller volume from the available extra. I took up Drobo on this offer. After a couple of days, it had done the necessary data reorganization and life was good.

The disk failure

One of the 1 TB disks failed early in the new year. Drobo became write protected again and Time Machine became unhappy. After a couple of days, Drobo had reorganized the data and forward error correction and again became read write. Magic. I didn’t have to do a thing other than leave the beast alone, a hard feat for a former Navy Nuke trained to do something when things go wrong! The something I did

Stop Time Machine
Unmount the volume
Tell Drobo to go to standby
Let Drobo do its thing unmolested

Replacing the disk

When I came back to replace the failed disk with a new Western Digital Caviar Red 3 TB disk, I recabled the drive on Fire Wire. The drive woke up on its own in healthy status. I inserted the drive in the slot of the failed drive. After a minute or so, Drobo discovered the drive and proposed creating an additional volume. I opted out of this. So here’s what I did.

Cabled the drive to the Fire Wire bus.
Added the Audio Interface to the bus as last device
Removed the bad drive from Slot 2
Installed the new 3 TB drive in Slot 2
Waited some
Started Time Machine

Time Machine is still sorting things out but has begun doing backups again. It will be writing data for a good bit of today. Once it finishes, I’ll unmount both volumes on that Drobo, restart the Drobo, and add volumes covering the new space. I should be able to create a second volume that is a full 2 TB.

What to do with the new space

Time Machine is aware of disks and slices. Time Machine is smart enough not to back up the backup volume. OS X is also aware of disks and slices and knows which volumes are on which disks and slices. Although there are new volumes out there, they should not be used for data because they are on the same disk as the Time Machine volume. That means they can’t be backed up. Death of the Time Machine disk is also death of these volumes.

Time Machine will let you add volumes to the backup pool. I believe these may be on the current Time Machine disk. So once I have Time Machine happy and the available 5.4 TB formatted, I’ll add the new storage to Time Machine’s volume pool.

Choosing Disks

Best Buy and the other local retailers have consumer grade disks on the shelf. Both WD and Seagate package disks with cables, screws, etc for retail sale as internal disks. The catches

Unknown handling by customers. Was one dropped? ++ungood
Designed for light use

I’ve always had poor luck with disks bought at retail. They seem to last a year or two before going tango uniform (toes up). So I prefer to buy from Mac Sales or Amazon who package disks properly (like Sun packaged its repair part disks). I’ve had good service with these, 4 years for the WD 1 TB that failed.

WD makes its OEM disks in several grades identified by color. The 5400 RPM disks are Caviar Red and Caviar Green branded. The Red are rated for small storage array use. The Green are rated for PC use. The difference is that the Red are designed for a bit more activity as the storage array may not spin the disks down as often as a PC will. Several things are at work here.

Multiple hosts can read and write to the array
The array can do read and heal passes over the data
The array is less aggressive about saving power by spinning disks down