Understanding the Difference Between Catalogues, Folders and Collections.

When you have been working with a programme for years you can sometimes forget how confusing some of the basic concepts of that application can be to someone just starting out with that programme. A case in point is Lightroom. I have been using the programme pretty much since it’s inception. At first I only really used it to catalogue my finished images, preferring instead to rely on Nikon’s slow but image quality superior Capture NX 2. When Nikon ditched upgrades to their app, I was forced to swallow the Adobe pill completely. Still, this has meant working with Lightroom in a limited or full capacity for the last 10 years. Sometimes, as my assistant reminds me, I forget that the way Lightroom handles files can be confusing and frustrating to many. The single biggest ‘issue’ I have come across is the confusion between Catalogues, Folders, Files and Collections. Here is a rough guide to work one’s way through the confusion.



First, some definitions:

Files - To all intents and purposes your images are files. An individual image is a data file, hence why every image has a file name.

Folders - A folder is the encompassing storage area where files are kept. For anyone used to working with a computer the concept is fairly simple. A folder is where you keep your files. With Lightroom, a Folder is where your original Files or images are kept. Subfolders are folders within folders such that you can have a tree diagram of folders. The important thing to remember is that folders are where you direct the original files. Confusion tends to reign thanks to the way that Apple handles files, essentially dumping them in a common area and requiring the photographer to sort them into Collections. The critical thing is that a Collection is not the same thing as a Folder.

An illustration of how Folders and Subfolders work. Image files are physcally moved into different folders

Collections - This is essentially a virtual or imaginary Folder. The Files DO NOT live inside a Collection, rather the Collection is like an index card system in a real-life library (the kind you walk into and smells of paper…remember those). The index card isn’t the same as the actual book that it denotes, it just directs the reader where to find the book. In the same way a Collection doesn’t hold the actual File, it just holds an index card of sorts which points Lightroom (or any other application that uses Collections) to where the File can be found. In summary, Folders are real and Collections are virtual.

An illustration of how a Collection works: the actual image files do not move, but are 'collected' into virtual folders or 'Collections'. Importantly, image files can be included in multiple collections while staying within the same folder.

Catalogue - This is where Lightroom stores all of its bits and pieces that make up the Collections, the image edits and the metadata that you have attached to your Files/photographs. Again, and this is important, the Files/photographs ARE NOT stored inside the catalogue. Saving the catalogue or the .LRCAT file is NOT going to save your photographs. The actual images or photographs are always separate to the Catalogue. Think of the Catalogue like a giant database of real-life libraries, where we can access which books are in each library. 

 

xmp sidecar file - An xmp, oe Extensible Metadata Platform, is a sidecar file (also known as a buddy file) that stores data that cannot be be directly stored on the original file format. In the case of Lightroom, all metadata and image editing parameters are stored on an xmp file which is kept either next to the original file (as in the illustration below) or in the resources of the Lightroom application itself. In order to save the edits made to a image in Lightoom the xmp sidecar file also needs to be saved or the entire Lightroom Catalogue (which holds the xmp files) needs to be backed up. 

Metadata - In a sort of circular definition, metadata is defined as being "a set of data that describes and gives information about other data." What this really means for image files is that metadata is textual data that is included to the original image file. You can access this data by looking at the properties of the image file, or by using a programme that can show the actual metadata (Lightroom, Adobe Bridge, Photo Mechanic etc.) Metadata includes information such as the time and date of image capture, captions, keywords, creator info, image editing parameters, copyright information and much more. 

How Lightroom Works.

Lightroom is what is called a PIEware (Parametric Image Editing) application. PIEWare applications edit the parameters of the image, not the image itself. This means that when you make edits in Lightroom the edits are actually stored as instructions that are only applied to the image on export. This is very different to a bitmap editor like Photoshop which make edits on the imagery directly.

In order for Lightroom to 'see' an image or File, that image needs to be imported into Lightoom's database or Catalogue. Importing does not actually mean that the image is physically moved into Lightroom. Rather, Lightroom creates a small jpeg copy of the original file (regardless of whether the original is a RAW or jpeg file) that behaves almost like a placeholder. When you look at the image in Lightroom, you are not looking at the original, but at the small jpeg. When you edit the image in the Develop mode of Lightroom edits are not being applied to the original file, but again to the jpeg surrogate. Only on export are the edit parameters finally applied to the original file so as to create a whole new version of the image. Again, the original image has not been changed in any way. The edits have been applied to the original file edit parameters to create a new file.

What this means though, is that when you edit an image in Lightroom, you are only creating an instruction. You are not actually adjusting the image itself. The advantage of this is that any edits made are not permanent and are non-destructive. The disadvantage is that there is another level of data that needs to be saved and stored. For the most part this data, in the form of the edits and adjustments made in Lightroom is automatically stored in the background files and resources of the programme. However, not understanding the difference between Catalogues, Folders and Collections can lead to accidental deletion or disassociation of image and edits.

The advantage of a PIEware programme like Lightroom is that the programme uses significantly less data when in operation than if the programme were accessing the original Files. By creating jpeg surrogates and edit parameters the overall size of a Folder of images is reduced exponentially. For example a folder of 100 RAW files from a Nikon D800 is approximately 6,7 Gb of data. In Lightroom the jpeg surrogates and sidecar .xmp files take up less than 40Mb. Even using Lightroom's Smart Previews (which allows images to be edited even if there is no actual physical connection to the original files - which are used to refresh 100% views) only increases this about twofold. The enormous advantage of this incredible size reduction is that the programme can crunch through huge amounts of images in significantly less time than if it were to use the original file data. A case in point would be searching through the Library for a particular image based on metadata (keyword, EXIF data or rating perhaps). In Lightroom I can search through an entire library of 157000 images in under a minute (with Catalogue optimised working on a 3.2 GHz iMac with 32Gb of RAM). Using Adobe Bridge, which accesses the original file data to perform the same task can take up to 15 minutes (using the same computer system as already mentioned). So essentially the strength of PIEware is in working with large libraries of images and in non-destructive editing.

The confusion arises when discussing Catalogues, Folders, Collections and Files when these have to be saved. If you want to save your original files onto a new hard drive you have to save the physical folder. If you want to save the edits you have made in Lightroom, you have to backup the Catalogue (file extension LRCAT). All the sidecar xmp files, jpeg surrogates and metadata is stored within the Lightroom Catalogue. It is possible to keep the xmp files alongside the original RAW files by either hitting Cmd+S/Cntrl+S (Mac/Windows) or by going to Catalogue settings (Lightroom>Catalog settings>Metadata) and checking the tick box that allows xmps to be written alongside the original files. Either way, this creates an xmp sidecar files that resides alongside the original folder. This means that the edits made in Lightroom are easily read by other Adobe applications such as Bridge and Photoshop and can be saved whenever you backup the folder that  holds the original files. Critically, saving the Catalogue (LRCAT) or Collections DOES NOT SAVE the original photographs! This cannot be stressed enough. The original files have to be saved separately to the Catalogue.   

A Suggested Workflow

My personal recommendation when working with Lightroom (note the ‘personal’ - this is not the same as ‘perfect’, but happens to work for me) is to place my original original RAW files into a chronological sequence of folders. I can do this ingest via Lightroom or something like Photo Mechanic. Either way the files are then imported into Lightroom. After working on the files I usually save the xmp sidecar files so that they reside next to the images themselves. This means that when I remove the images from my Lightroom catalogue all the edits and metadata are kept with the original file (as in the image above next to the definition of xmp sidecar files). 

You can set preferences so that xmp files are written directly to folder where the original image is kept.

I use Intego Backup Pro to backup all my RAW files so that I have two hard drives (as in current work hard drives - I also archive work that is completed) of which one is a clone of the other. This creates a backup of the original files as well as xmp sidecar files with the edits. I then have my actual computer on Time Machine so that the Lightroom Catalogue is regularly backed up. This means that both my original files as well as my Lightroom Catalogue are backed up on different devices.  

Hopefully, this has cleared some of the confusion regarding Lightroom catalogues and the images that are ingested. If you have questions, please drop in the comments section or email and I'll try and respond.