I’m in need of some guidance when it come to backing up files which are often changing.
For now I have different backups configs running different days of different data.
User files, DB backups, media files such as photos, videos and sound recordings.
I specifically have some questions about backing up certain photos and of them the JPEG/DNG/TIFF and XMP sidecar files.
As my asset management system writes out XMP metadata to its own file but also directly into certain filetypes like jpg, dng and tiffs this filetypes constantly changes as soon as I add a label or change rating etc. And this might be done to a huge amount of files often.
The original RAW files are not changed all the time. Neither are video or sound recordings. But XMP sidecar files for them will change.
How would you recommend me setting up the backup and retention for those directories and files so that I have a good reasonable balance between back-up storage consumption, backup times etc.
Today I’m running Duplicati 3 times per week for these directories with 50MB upload size and I’m keeping the backups “Unlimited”.
The XMP and sidecars are a few KB only and there one of two of them.
The RAWs are 12-30 MB each.
The jpegs are about 5-15MB these are both generated directly in camera as well as being finalized and exported from RAWs by me.
Should I back up separate based on year instead? of running one huge set for all medias?
Backup less often?
Lower datablock size?
Change retention?
I could turn off the writing of XMP data into the files as well although I would like to avoid it.
It’s tough to suggest things in situations like this as results can vary greatly depending on how the metadata is written to the image file.
If the image / video labels and ratings always live in the same blocks of the files (so they not grow not shrink n no matter the values) then Duplicati will only back up the small changed blocks.
However, if the metadata is store before the end of the file and causes subsequent data to change “location” then the updated data AND the following moved-but-not-changed content will be backed up, causing longer backups and more backend storage needs.
https://en.wikipedia.org/wiki/Extensible_Metadata_Platform#Location_in_file_types was tempting, but it soon began describing in file format internals. Maybe you know them? Or perhaps experimenting with a file would be easier? If whatever you change causes Job --> Show log --> Remote (and click on the dblock) to show a dblock of about full image size uploading, then your change likely shifted image contents, thereby defeating deduplication. Another way to find the amount of upload that the change caused is by Job --> Show log --> General. Example of where to look:
If you are lucky, the changes you usually make will be at the end of the file, in which case Duplicati will scan through early blocks of the file, find that they have not changed, not upload them, and only upload the end. Editing the actual image is likely to cause widespread changes, but embedded XMP information is unknown.
