I’ve been playing around with some of the options in Duplicati to get to know it better. One I’m interested in is the prefix option. At first, I set it to something non-default. Now that I’ve been using Duplicati, I have something specific I want it to be but can’t successfully change it.
From another thread, I see that folders are the preferred organization method vs prefixes. I’m wanting to use both. I still have them in separate folders, but also have a desire to set the prefix, as well.
Here is what I’m doing:
Setting the prefix option in the Duplicati settings (I want all backup jobs to have the same prefix). For example, “INITIAL”
Create and run a backup job. The backup files now have the INITIAL prefix.
Change the prefix in the Duplicati settings (same seems to happen in the specific backup job, though). The new prefix could be “FINAL” for example.
Verify the backup job’s files. This will fail due to missing files (looking for the new prefix, FINAL, and there are only INITIAL files there).
Change the prefixes of the actual backup files from INITIAL to FINAL.
Verify the backup job’s files again. This fails again.
Run a database repair. This fails terribly in that it deletes my backup files.
I’ve tried a few combinations of changing the prefix in the Duplicati settings, the backup job settings, both, doing DB repairs vs recreating the database. Nothing seems to get the backup job to recognize the altered prefix files as its own even though there are no other files in that backup job’s destination folder.
I suspect the database knowing the actual names of the files makes FINAL ones unknown, thus deleted.
Changing the prefix option doesn’t change existing DB entries AFAIK, so changing to final and backup:
Recreate says:
Rename of all destination files to fix prefix gets this on Recreate:
Errors 2
2020-09-07 16:37:04 -04 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b300cfa9daca94cbb8aa080dd3055eba0.dblock.zip.aes by final-i42131a6bfbc441ec948094c601313aa4.dindex.zip.aes, but not found in list, registering a missing remote file
2020-09-07 16:37:04 -04 - [Error-Duplicati.Library.Main.Operation.RecreateDatabaseHandler-MissingFileDetected]: Remote file referenced as duplicati-b6901176f73984214985f8ca22f585a5c.dblock.zip.aes by final-i76fa670166f044b393ee4814a6455509.dindex.zip.aes, but not found in list, registering a missing remote file
This is because the dindex file contains the (complete) name of the dblock file that it’s associated with.
Duplicati Repair will regenerate missing dindex and dlist files if it has the data, so delete all the dindex.
Run Repair then expect new dindex files. At this point, you’ve probably finished (awkward) conversion.
Repairs delete all my files. Even after deleting the dindex file. A repair is deadly.
However, the recreate (delete and repair) works like a charm.
Here’s what I just tested as successful:
Run a backup with an initial prefix (e.g. “INITIAL”)
Change prefix in duplicati settings (e.g. to “FINAL”)
Rename files to new, “FINAL” prefix
Delete dindex file (assuming if there are more than one, all should be deleted)
Recreate the database (this option is for the local DB) = Success, no errors, all files still in destination.
Run another backup = Success, new files have the “FINAL” prefix.
Run a restore of a file from a backup that was done with the “INITIAL” prefix = Success
Changing step #5 to be a repair of the database causes all files to be deleted in remote destination. Sad panda. But the recreate worked like a charm. Important but simple mistake/difference.
Why doesn’t a repair do the trick? Why only a recreate?
You seemingly skipped the rename files and Recreate step. After that, the DB knows new file names. Without that, the DB has the old names. so you’re back in your previous problem with Repair deletes.
Recreate takes whatever the destination says. Repair uses DB records, so incorrect ones confuse it.
Your sequence might also work, but be careful about what’s being downloaded, e.g. watch the lines in
About -->. Show log → Verbose during Recreate, or see if progress bar gets beyond 70% to the right.
There are three potential phases of downloading dblock files, and last 90%-100% one downloads all…
I should mention that this whole procedure is obviously beyond what the UI provides, and possibly beyond original design target. Manual changes on the destination are usually something risky and to be avoided…
If you make a long-lived valuable backup, and change Duplicati to a new version, a test on a small backup would be useful before running the procedure again (if that’s in your use case), to be sure things still work.
I’m not sure what your method does with dblock files, but I tested my rename-and-recreate and it got none.
I’m not sure what you mean? Step #3 and step #5 in the process that worked to successfully change the prefix above are the rename files and recreate steps.
Recreate takes whatever the destination says. Repair uses DB records, so incorrect ones confuse it.
Makes sense. The deletion that the repair does seems so drastic, though. Kinda surprises me.
possibly beyond original design target
This definitely seems beyond original design intent. Hacking is fun, though.
Although, I do wonder why there are any prefixes to begin with. Using one folder per backup job seems to be the simplest, so why is there a need for a default prefix at all? I would have assumed the default would have been a blank prefix. Blank would actually have been my first preference.
I’m not sure what your method does with dblock files, but I tested my rename-and-recreate and it got none.
Not sure I understand. There’s no manipulation of the dblock files other than the manual prefix renaming (step # 3). What did you get none of? What exactly is your rename-and-recreate process? Does it differ from the one I posted on 2020-09-07 21:25 ET that was successful for me?
Linking to that from here will provide another scenario that maybe can be worked in if a fix can be done.
I was actually pretty impressed by some of the Duplicati messages I saw where it seemed to guess at what the user might have done with the backup prefix. Maybe it will someday detect an all-files rename, however maybe someday an enhancement can be made to make this less do-it-yourself, and so safer.
Feature requests are way ahead of available developers, and beyond that, demand can also factor in…
You’re very much encouraged to learn your way around, maybe hacking code, or at least helping people.
Forum helpers almost always suggest separate folders. I’m not sure what the need for prefixes is. Even destinations that don’t really do true folders (e.g. cloud storage buckets) typically can do pseudo-folders.
No dblock file downloads, which can be a huge time and cost problem on a large remote backup. When Duplicati does a Recreate it needs to find all blocks referenced in a dlist file, and know dblock block is in. Processing a dindex file gets this. If a dindex is missing, an exhaustive dblock search may be required to locate the particular dblock by its hash. Feel free to look in some .zip files to see how referencing works. How the restore process works is also useful. Possibly your slightly different plan (still with Recreate) is fine, however I wasn’t sure how Duplicati would map blocks to dblock files after you deleted all dindex…
I have made a portable application out of Duplicati windows install and would like to change prefix for backup files (duplicati by default) permanently to e.g (myname) in my portable application so the user would not require setting up the option in Settings / Default Options anymore and that would also not depend on local AppData.
Can someone help, i have been trying to find but no luck!
