Existing archives in destination check?

Features
1

Is there a reason destination files aren’t all prefixed with something unique-ish like the name of the local sqlite db file the backup?

Something like that would squash most of the “remote files not in the local database” issues related to sharing destination folders and would simplify manual maintenance of files…

2

Don’t think you’d want to use the DB name as that can change, thought about using backup name as the prefix but again would be cumbersome if you renamed the job.

Maybe ask for a job prefix on the backup job setup and ensure it’s unique across jobs? Won’t help if you have multiple sources all pointed at the same place though.

Also this defeats one of the strengths of Duplicati which is you can loose everything and so long as you have the remote files you can restore. You can still restore if the prefix is non-standard but would be an extra hurdle that might confuse the less tech savvy.

Maybe it’d be better to have Duplicati check for existing files when setting up a new backup job and prompt with a warning if it finds some? “Warning: It looks like there are already existing files in this location, are you sure you want to continue etc. etc.?”

1 Like
3

That was going to be my follow-up comment after hearing if there was a coding reason for how it’s done now (files-per-folder limits, for example).

How can the DB name change?

4

Without the prefix duplicati- a lay person may get confused as to why there are so many weirdly named files in their Cloud Drive and even delete them unknowingly. The prefix could be extended to add a unique backup id like so - duplicati-abc012-

5

I agree with @sanderson’s and @mohak 's concerns for not keeping the current naming convention, but I think @sanderson’s restore issue is a key reason for keeping things the way they are now.

Something other things that come to mind:

  • If you use the backup name (or another keyword) as prefix, you will sacrifice some security, because the filenames reveal some information about what’s inside. Random names will never give any clue what type of data the files contain.
  • Storing multiple backups in the same folder should be discouraged in any way, but using unique prefixes should facilitate this. I suppose a better way to avoid conflicts is adding a check if Duplicati backup files already exist in the destination folder when creating a new backup job.
6

I agree with pretty much everything being said - but I think I’m not making myself clear, so I’ll try again with some examples. Please let me know if this is actually what you all got from my original description.

  1. Yes - having a destination check for existing duplicati- files is a good idea.
  2. Yes - encouraging separate folder for each backup is a good idea, but it doesn’t always happen (my guess is usually not on purpose).
  3. I’m wondering why we can’t use the name of the local sqlite file already created by Duplicati (apparently with a random name) as a distinguishing feature of the remote files. For example, locally I have AAAHFSVBZF.sqlite so remotely I would have files like the following, all obviously related to the AAAHFSVBZF backup:
    a. Duplicati-AAAHFSVBZF-20171016T024101Z.dlist.zip.aes
    b. Duplicati-AAAHFSVBZF-b00ec9bc9ab53459ca77b81f6fd418ef3.dblock.zip.aes
    c. Duplicati-AAAHFSVBZF-i0a2fd13f21414b5838cd007193217559.dindex.zip.aes

In my mind this would:

  • simplify cleanup when deleting jobs (just delete all files starting with Duplicati-AAAHFSVBZF-)
  • support multiple backups in one folder without needing user action to set a backup name
  • allow for easier identification of “all” related backup files (local and remote) in instances where manual maintenance (such as moving destinations) is necessary

Oh, and just because I bumped into this so recently after replying to the above… :smiley:

7

Duplicati only works if it has a correct “cache” of the remote folder. When files appear in the remote folder that were not previously there, Duplicati needs to recover from this. If we add a random prefix to the filename, this will not fix it.

We could use an “auto-repair” when discovering new files. Currently, Duplicati attempts to delete them, but it could just as well download them and add them to the database.

But I don’t see the use case for this. The user should never modify the remote folder, so the only case where this should happen is if the user deletes the local database. I think having some manual intervention in this case is a minor inconvenience.

But we should change a the error message for this situation to explain what has happened.

Both the database name, and the backup-job name can change. The database name is randomly generated on the first backup.

If we add more to the prefix, it really is the same as specifying a custom prefix. And this does not work great, because if different backups are found, Duplicati won’t know which one you want to work with.

I agree. Not sure how to help with this. If we have a random name (say from the database), a user could end up with multiple backups in the same folder.

If the machine breaks and the user needs to restore, there is no easy way to guess if the important files are using AAA, BBB, or CCC prefix. So I think this moves the problem from the setup phase to the recovery phase.

8

Well that takes care of my thought then as I was assuming the database name was immutable. :slight_smile:

I do still think an “hey, this folder already has archive files in it” check & message would be good.

9

Apparently it was just a dream that I had already implemented it:

10

I don’t suppose your dream included actual code that you could just type up IRL? :slight_smile:

11

Oh sure, take his word for it instead of mine. Just because he wrote the application :smile:

12

Heh - well I just assumed I did such a poor job of describing what I was imagining that you misunderstood me. :blush: