After deleting and recreating database, backup is 'opening' files un-necessarily

Support
1

Hi,

I recently had to rebuild the database for one of my backups, and since then the backups are taking a lot longer than they should. I’ve looked in to it, and Duplicati seems to be ‘opening’ files even though they haven’t changed in a long time. It’s ‘opening’ 106 of the 551 files in the backup set, even though I ran the backup, and then immediately ran it again without any changes being made to the files being backed up.

If I use the live log viewer with ‘profile’ selected, I see entries like:

Checking file for changes , new: False, timestamp changed: True, size changed: False, metadatachanged: False, 9/4/2020 1:07:47 PM vs 9/4/2020 1:07:47 PM

Assuming that those two timestamps listed at the end are Duplicati’s ‘remembered’ timestamp, and the actual timestamp of the file (they certainly match the latter), you can see that they’re identical. However, it is still considering them different.

I’m running 2.0.5.1_beta_2020-01-18, which I believe is the latest beta.

Any thoughts?

Thanks

Andy

2

Yes, I have seen this and I know there was at least one discussion about it on the forum.

It may be intentional just to ensure all your data is properly backed up. (Or maybe it’s not intentional, I’m not sure.)

In any case you’ll find that subsequent backups will be back to normal speed.

3

That’s the thing, it doesn’t. Subsequent backups are also doing it.

It’s as if the comparison is being done with too much precision, and the database version of the timestamp isn’t being updated.

Andy

4

That’s exactly right, from what I recall. After a database recreate the timestamps have low precision and Duplicati notices that they don’t match the higher precision it reads from disk so it forces the file to be reprocessed.

Interesting that subsequent backups still do it. I haven’t seen that myself. Let me try some testing…

5

Ok I was able to reproduce your problem (subsequent backups keep doing the full file scan). I think this is caused by Duplicati discarding the backup version if nothing changes (no file adds/deletions and no file content changes). Metadata only isn’t enough for Duplicati to keep the most recent backup, it seems.

If you add or remove a single file or edit a file’s contents, it is enough for Duplicati to keep the most recent backup and then to stop this full file rescan on the subsequent backups.

I think Duplicati should retain the most recent backup even if it’s only metadata that has changed.

6

I agree.

So are you saying adding a single file to backup will ‘fix’ this behaviour?

Thanks

Andy

7

Yes. Any simple change like that (beyond metadata) will be enough for Duplicati to not discard the backup job.

8

Ok, created an empty file and run a couple of backups. The first took over an hour as before, but the second completed in a matter of minutes.

Thanks for the help, but definitely not the behaviour I’d expect.

Andy

9

Yep I don’t think this is correct behavior either. If you are willing, please post an issue to the Github project page.

10

I’ve done that @drwtsn32:

That enough detail do you think?

Thanks again for the help working out what was going on, and suggesting a way of working around it.

Andy

1 Like
11

Yep that looks good to me! Thank you