Taming the Photo Image Catalog Problem

The Dismal Retired Moocher has been taking photography since high school and digital photography since the early Oughts. His first camera was Dad’s Argus C3 bought when we moved to Gales Ferry, Connecticut. Dad didn’t use it much so it went to college with DW.

DW’s first digital camera was a Sony point and shoot styled like a range finder camera. It had a built in flash, optical finder, and a bit of a zoom lens. Later, a Panasonic Lumix point and shoot replaced it. Then a series of iPhones and iPads, and Sony mirrorless DSL cameras. And a sequence of photo editing and managing applications, Apple iPhotos, Apple Aperture, Adobe Lightroom, Phase One Capture One Sony Edition (retired by Phase One), and most recently Skylum Luminar in its various editions.

So we have a lot of files in a lot of places dating back to the beginning of digital photograph time here at Dismal Manor. It became a mess with multiple copies of everything as Apple struggled with iCloud’s hierarchical file system and the iOS services that used it. And various MacOS updates forced photo product updates as Apple had “better ideas” and migrated from one product to another in a forced photographical death march and DW flirted with various “serious photography tools” like LightRoom and CaptureOne.

This post is about our journey and how it led to some of the workflow I now use and the photographical computing problems that they solve. In particular we look briefly at CYME’s PeakTo product. The more you fiddle with it, the more it amazes

Revisions

• 2023-04-22 Original

References

1. https://cyme.io/mission/, CYME Mission Statement introducing Avalanche and PeakTo.

Digital imaging at Dismal Manor

Over the century, DW has owned a number of digital cameras and used a number of photographical software products. This is a rather lengthy journey from simplicity to complexity and back to simplicity and through increasing levels of chaos. I say digital imaging because this journey is both a hardware journey and a software journey. It is anchored by MacOS as DW took one look at Windows OLE and ActiveX and immediately grasped the security implications. He held off on home computing in a big way until Steve Jobs returned to Apple with his FreeBSD based NeXT OS to launch MacOS and the modern Macintosh product line. During this period, DW has had a Mac Tower, an Intel Mac Mini, and currently, an Apple Silicon iMac.

In the beginning Sony had a go …

The first digital camera DW encountered was a Logitech thing that a visiting Finn brought with him for a summer internship at ABB where DW was “working” as a simulator hacker at the time. The second was a Sony Mavica that Cousin Kenny brought to Thanksgiving Dinner. Eventually, DW bought a Sony at Christmas 2002 for holiday snaps etc as film camera had died in storage. This was our gateway. In November of 2002 DW had bought an early Mac desktop box that came with Apple iPhoto among the included software kit. DW used the two to make some early photos of Lance and Newt, a few of which may have survived to the current era.

Enter the iPhones

In 2007, Apple introduced the iPhone and DW bought an original after the category launch feeding frenzy died down. Now you could have a camera in your pocket but not in the Navy Warfare Development Command battle lab where DW was working at the time. We took a lot of pictures with it around town and our earliest dog pictures. Pictures from this era are split between a Panasonic Lumix that DW still has and the iPhone, horid as it was and all ended up in iPhoto.

Enter the Sony DSL Mirrorless Cameras

In 2011 DW bought his first mirrorless DSL camera from Sony. DW has small hands so wanted something on the smaller side. He also wanted simple menus yet robust sensors and lenses for around home snapshotting. He settled on a Sony A65 for his first and later acquired a Sony A6300 that is our current camera.

And the Digital Lightroom Stuff

In the beginning, there was iPhoto. I used it quite a bit around home and in Newport, RI while I was on exile to BoSox (Boston Red Stockings) territory. After a while Aperture came along and I took a tumble with it. Then Sony and Phase One offered a gateway drug version of Capture One for free and a $99 version that was full featured but came with only the protocols to talk to Sony A-series product. Phase One went on to make similar offerings for Fuji and Panasonic and after a few years lost interest and discontinued the versions as nobody was moving to the full, all cameras product. Or buying Phase One cameras.

Then we flirted with Adobe LightRoom between Aperture and Capture One. Finally, once iCloud photo library was sorted, we settled on Apple Photos App with editing plug-ins. And that’s where we are today. Every time DW switched photo editors, there was a project to migrate the image catalog from the old one to the new one. Oh and everybody had followed Adobe’s lead and started putting the data and original images in proprietary application-specific databases. So they could hold your images at ransom.

And each editor had its own way of doing “non-destructive” editing. That is each editor records markup that describes how to transform the raw image into an edited finished image. The best of these let you make multiple derivative images and track each individual rendering for you.

And each editor had its own scheme of tagging and organizing images so you could find them again. Most were oriented to working pro who are client, project, and session oriented.

And there was the race to “the cloud” and to schemes to milk the customers a little each year.

And each editor had its unique techniques with associated unique sidecar file annotations that were unknown to the others. So some image transformations could be lost.

So this is photographical bliss, right?

Over the years, the following issues developed.

• Do we keep images forever? How do we preserve them? How do we ever find one again?
• What do we do with them? Post them on Mastodon? Make holiday cards of them? Who even sends holiday cards in the roaring 20’s?
• Do we edit the images? or just keep the awkward camera originals around?
• If we use non-destructive editors, how do we recreate the edited image for a new application?
• If we use non-destructive editors, how do we preserve our work product? How do we recreate it if the tool is discontinued?
• How do we archive the original camera raw files for the National Archives?

I wont’ promise to answer all these questions but I have found answers to the backup issue and editor migration. The backup solution is OS neutral. The catalog migration issue has an Apple-specific solution.

Apple iOS Stuff

Apple has done a creditable job of developing a photographical system designed for casual use and image sharing but also friendly to professional videographers. iPhone Pro takes much of the footage you see on YouTube. And probably half of the images seen on social media. Apple has made it easy to export images to popular services. Apple has also made it easy to edit images on camera. Apple automatically geo-tags images, does facial recognition and people tags images, and does automatic object identification of plants and creatures. Apple MacOS Photos makes it easy to review and fix machine learning goofs and to automatically refine and manually adjust image appearance.

Apple appears to be stable with the current Photos App and associated image processing and image recognition libraries. I suspect some of this is in self-defense as Apple doesn’t want your child porn and dick pics arriving on its servers. Google is also doing ML screening of images sent to them. Each is doing what it thinks is needed to enforce its acceptable use policy and US law. Apple is not passing suspect images up to iCloud or letting you E-mail them at your end.

So, Dismal Manor has seen iPhotos, Aperture, LightRoom, and Capture One come and go. Happily, Apple has always provided a migration tool that carries over your originals and whatever you did in the obsoleted tool. Apple will bring along your sidecar files created by editing plugins and returned to Photos for preservation. Most editing plugins will return a finished TIFF image and some but not all, return the sidecar file.

Adobe Showed the Way

Adobe designed LightRoom and Photoshop to work together. LightRoom brought images off of media, rendered previews, filled in metadata like client, job, session, date, geotags, copyright owner and license grant, etc and provided tools to refine the image. Once “developed” in LightRoom, Photoshop let you do the creative stuff like remove Aunt Millie’s liver spots and snip off Denis the Menace’s cow lick. Adobe wanted to use one from the other so it came up with the Photoshop Plugin API that let one Adobe photograph tool load the other as a library and call services in it. And Adobe developed a protocol for passing the original image in and receiving the finished product and the editing instructions.

Our Current Workflow

So what does Dismal Wizard do to bring order to the chaos? He uses a number of stock Apple tools plus some third party tools for selected tasks. Most of these tools are designed for an Apple environment but some like Mylio and Skylum Luminar are also available for Windows.

All in a day’s work

DW’s current workflow is to shoot with whichever camera is at hand. Each day, he imports the images into Apple Photos, edits the images with Skylum Luminar Neo using the “Forest Stream” preset. He then reviews the results, reviews the edit stack, and makes adjustments as needed.

Image Preservation

After everything is in Photos App, Mylio grabs it for backup on the Mac’s internal disk. From there Time Machine grabs the Mylio images directory and spools it to the active Time Machine stores. We have 2, one on Mr. Peabody, our TrueNAS file server, and a second on a LaCie 8TB USB disk. Mr. Peabody replicates the files to Sherman, our backup TrueNAS server. Internally, Sherman makes a copy to an 8TB internal volume used only for backup storage. The LaCie can be grabbed should we have to evacuate.

In addition, we subscribe to BackBlaze home backup for $72/year and a copy goes to BackBlaze.

Mylio does a fair bit of cataloging and organizing on its own. Among the things it does are

• Grouping images by date on a calendar grid
• Face recognition and person tagging
• Duplicate recognition and clean up
• Edit in Apple Photos option
• Smart folders, etc.

Image Usage

In the old days of 35mm, DW took slides. And DW schlepped a slide projector and screen on family holiday for show and tell. Today, DW posts selected images on Mastodon where we have presences as @DismalManorGang@mastodon.online in Germany and @DismalManorGang@cupoftea.social in the UK.

Every day, we post one or more photos, usually accompanying some story about life with #Greyhounds or things happening around Dismal Manor.

Usually, we locate images in Apple Photos App, copy them, paste them into Mastodon, and then write the text. When I feel it is useful for an image, I’ll add ALT text for the vision impaired describing the key elements of the image as they pertain to the post message. Usually, the image is there to catch eyes and influence people to stop to read the post. Usually, they are eye candy unless a dog was caught in the act or some drama happened near the Manor.

Image Organization

DW currently uses CYME PeakTo to organize images. PeakTo is an evolving product that uses the Apple Machine Learning stuff to tag your images, do facial recognition, etc. And it can search on the tags, make smart folders using tags and other metadata, etc much as Adobe Lightroom did. But the big thing it does is the bulk of the tagging and data entry. In a coming version, it will add owner information, copyright information, and license grant information, usually CC-SA-NC with attribution. So this is what PeakTo will add to each image it catalogs.

PeakTo provides the usual favorite, tagging, colors, and stars and can search on these or on EXIF or IPTC metadata. It is also aware of derivative images and groups them with the original.

PeakTo can edit in Apple Photos. Apple Photos then edits in Skylum Luminar Neo and returns the stuff to PeakTo to catalog. So I can locate an image and re-edit it for a new application.

PeakTo Talents

CYME designed PeakTo to solve the multiple sources and formats problem. PeakTo is able to read the following image catalog formats

• Apple Photos
• Adobe LightRoom
• Phase One Capture One
• Skylum Luminar
• Google Photos
• Apple iCloud Photo Library
• Apple Aperture
• Flickr

CYME Avalanche, a companion product, converts between catalog formats for those products above that have a local catalog.

So, PeakTo solves a very big and painful problem, the need to convert from one catalog format to another. It dodges this problem by locating and cataloging each image in each product catalog, building thumbnails, and tagging with subject tags. If I had had PeakTo available, I could have built the common catalog of my Apple Photos, iCloud photo library, Aperture library, LightRoom Library, and Capture One libraries. I would not have had to export and reimport and re-catalog my images with each change of image management and editing tools.

The screen capture above shows the PeakTo main window. Image sources appear to the left. Selected image data appear to the right. An image film strip is at the top. And a full frame preview is in the central view. Up at the very top, you can apply filters using date, tags, stars, colors, hearts and flowers, etc. At the left you can create folders from selections and smart folders from search criteria.

When cataloging images PeakTo

• identifies objects appearing and tags the image as containing them. For example, one image contained “dog” and “book”
• identifies people and tags them
• Groups into folders
• Supports user tagging, star ratings, color markings, favoriting manually entered.
• Groups into smart folders by tags, date range, ratings, colors, favoriting, etc.
• Fills in owner, copyright, license grant, commercial stuff like client, job, shoot, etc.
• Locates master images
• Locates derived image products
• Finds screen shots and stuff so they can be culled.

A couple of examples of the classification AI at work appear below. On the left is a picture of Rocky Greyhound in the yard which is correctly tagged. On the right is a photo of a dog and a cat. It is unclear which, Rocky or Missy, the AI thinks is a cat. So review the automatic tagging.

Automatic categories are useful. The AI will bin a screenshot correctly and filters can group them to locate images to illustrate a blog post or to clean up social media chat images of long ago weather radar.

These things should look familiar because most photo management and editing products have similar capabilities. The thing that is unique about PeakTo is that it can wrangle multiple image sources in multiple formats from multiple vendors in the currently favored manner. This capability saves all that image recataloging and resulting duplication. In cleaning up the duplicates, I broke all of my earlier catalogs. Anything not imported into Photos is now lost. But some images were tagged as originating in iPhotos and PeakTo finds those. PeakTo is a very powerful with many capabilities such as this one to be discovered by exploring the user interface.

In this example, brought up by a random click, PeakTo is looking for brown. It found Rocky copping a kip on the somewhat bigger bed, then found other images of Rocky in repose. A couple more random clicks found this set.

Avalanche

Avalanche solves the different catalog formats problem. It converts between catalogs. From LightRoom to Capture One or Luminar. It preserves most but not all of the edits. But it can get you out from under paying license fees should you need to rework an image in the future.

Avalanche and PeakTo share the catalog reading libraries. It should be able to exfiltrate images and marshal them for you in the catalog of your current favorite. Avalanche comes in versions converting many input formats to a favored output format or converting between multiple formats.