Needless and Repeated Backups

Hi Folks

I’m using Duplicati
I’ve noticed that I have two folders whose contents are always backed up, even though the contents have not changed. The reason I see is that for all of the files in question, their modification date is earlier than their creation date. This has come about for two reasons (1) the files were restored from cloud backup by a gaming platform (game save files), and (2) files were restored from a local backup created last year by a genealogy program. I have now excluded these two folders as an interim measure but of course, would prefer if Duplicati handled these files properly, as expected.

Having a mod date earlier than the create date would not seem to be unusual. Will Duplicati properly handle this situation in the future? It is important because unless one notices the repeated backups, there is no other way one would know of it.

There are utilities that can be used to change these dates but, again, unless you know of the situation you won’t know. Thoughts and advice would be appreciated (I’m a new user).

Thanks … Noel

What clues did you see (the more extensive the list, the better, because I think this is actually hard to see).

Block-based storage engine and How the backup process works would be worth reading if you haven’t yet. Basically Duplicati is supposed to only upload file blocks that aren’t already backed up, otherwise, it simply points to its old blocks. The file appears in the dated backup tree whether or not anything actually changed.

You can find out how many files it saw as candidates for checking-for-changes in Show log of the backup, e.g.OpenedFiles: would probably count all files in your two suspect folders, if Duplicati checked their files.

One can probably find some names behind that count by logging at Verbose or higher, either using live log at About --> Show log --> Live --> Verbose, or using –log-file with –log-file-log-level=Verbose. Example line:

2019-05-01 15:14:00 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\backup source\length1.txt, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 3/3/2019 11:29:09 PM vs 1/1/0001 12:00:00 AM

which in this case looks like a new file, but these are the sorts of tests that decide whether to open/backup.

If there is a problem where these folders are always backed up, a test for that would be to create a backup containing only these two folders and run it twice. If issue exists, it would upload twice and take same time.

Result for “no issue” would be no backup, because by default a new “view” of files isn’t done if no changes. –upload-unchanged-backups in Advanced options can override that and get a view, but can’t force uploads.


C:\backup source>perl -e “utime(time,time-24*60*60,‘modified_before_created.txt’);”

Start of first backup, which uploads files:
2019-06-15 12:13:57 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Operation.BackupHandler-BackupMainOperation]: Starting - BackupMainOperation
2019-06-15 12:13:57 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\modified_before_created.txt
2019-06-15 12:13:57 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-CheckFileForChanges]: Checking file for changes C:\backup source\modified_before_created.txt, new: True, timestamp changed: True, size changed: True, metadatachanged: True, 6/14/2019 4:09:19 PM vs 1/1/0001 12:00:00 AM

All of next backup run, which sees no work:
2019-06-15 12:14:06 -04 - [Profiling-Timer.Begin-Duplicati.Library.Main.Operation.BackupHandler-BackupMainOperation]: Starting - BackupMainOperation
2019-06-15 12:14:06 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\backup source\modified_before_created.txt
2019-06-15 12:14:06 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FilePreFilterProcess.FileEntry-SkipCheckNoMetadataChange]: Skipped checking file, because no metadata was updated C:\backup source\modified_before_created.txt
2019-06-15 12:14:06 -04 - [Profiling-Timer.Finished-Duplicati.Library.Main.Operation.BackupHandler-BackupMainOperation]: BackupMainOperation took 0:00:00:00.003

No second backup made. Can you supply details on how to get this, or was it maybe a misunderstanding?

The mechanism is already in place to handle this. The default behaviour that is used by Duplicati in identifying files is fname+mdate+size. If these 3 match to what’s already in the database the file is skipped irrespective of its content. This satisfies the requirements of most users. In your case even if a file is restored and it has already been backed up then Duplicati will hash the file and possibly contents of the file and since it already has the information in the database it will only modify the expiration date for blocks and not upload any data since the data is already backed up.

To change this behaviour you can use the -disable-filetime-check option. This will make Duplicati hash check all files. A mismatch on of the hash will cause Duplicati to start looking at the contents of the file and upload any changes.