Deletion of older Backup data - is this how it should work?

Support
1

Hello all,

I’m using Duplicati since some time. So far only smaller hickups, nothing special.

Now I came across the following situation:

As part of a move from a hard drive (HDD) to a solid state drive (SSD) installation of my Windows system, I enhanced an existing backup to include also folders with music and video data, about 65 gigabyte. Then I ran this backup job. The added directories were saved to the Duplicati backup (took more than six hours).

Move to the SSD worked fine. Duplicati up and running in no time, because the HDD was cloned to the SSD, only without certain data, like the music and video data, to speed up the clone process.
Now the Duplicati backup job started before I restored the music and video files, therefore the according directories on the SSD were still empty.
During the backup Duplicati stated “deleting unnessary files”.
The outcome of the backup job was that all backup versions that contained the music and video files were deleted!

No I wonder what kind of backup is this were data that is not still present on the source system gets deleted from the backup?
Not only for complete directories, but also for single files.

Maybe I understood or setup something wrong, but this creates some serious doubts about the backup concept of Duplicati.

One important inforomation, though. The option “Backup retention” is set to “Smart backup retention” like it is offered in the GUI.
I’m using Duplicati 2.0.6.1_beta_2021-05-03

Best Regards,
Ansgar

PS: Fortunately the data is not lost for good, because I still have it as a manual copy on some other external drive.

2

What are your backup retention settings? (On page 5 of the backup configuration.)

3

Hello,

the retention settings are as follows (standard setting from the GUI):
--retention-policy="1W:1D,4W:1W,12M:1M"

Here some more information from the log:

"DeletedFiles": 9588,
"DeletedFolders": 23,
"DeletedFileCount": 2514,
"DeletedFileSize": 65899431994,
"FilesDeleted": 2516,

"DeleteResults": {
    "DeletedSetsActualLength": 2,
    "DeletedSets": [
      {
        "Item1": 1,
        "Item2": "2021-05-12T10:13:58+02:00"
      },
      {
        "Item1": 12,
        "Item2": "2020-05-13T14:55:54+02:00"
      }

"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed",
"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / 1.00:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00",
"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 12.05.2021 10:13:58, 11.05.2021 17:18:52, 28.04.2021 17:07:06, 09.04.2021 16:07:08, 09.03.2021 14:49:10, 02.02.2021 13:18:37, 14.12.2020 14:41:09, 13.11.2020 10:22:30, 20.09.2020 14:57:46, 17.08.2020 12:34:32, 13.07.2020 21:26:22, 13.05.2020 14:55:54",
"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: 13.05.2020 14:55:54",
"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 12.05.2021 10:13:58, 13.05.2020 14:55:54",
"2021-05-19 10:41:21 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteRemoteFileset]: Deleting 2 remote fileset(s) ...",
"2021-05-19 10:42:09 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20200513T125554Z.dlist.zip.aes (527,92 KB)",
"2021-05-19 10:42:09 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20200513T125554Z.dlist.zip.aes (527,92 KB)",
"2021-05-19 10:42:09 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-20210512T081358Z.dlist.zip.aes (1,67 MB)",
"2021-05-19 10:42:09 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-20210512T081358Z.dlist.zip.aes (1,67 MB)",
"2021-05-19 10:42:09 +02 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: Deleted 2 remote fileset(s)"
4

Ok, I think the problem is that your music and video files were only in a single backup version. According to your retention policy, during the past one week it will discard backup versions if more than 1/day exists (to get it down to 1/day maximum). After that, during the past four weeks it will discard all but 1 version per week. And so on…

Files that are not on your system for at least as long as the “Y” timespan in each “X:Y” interval pair are at risk of being deleted from backup when the backup snapshot versions are pruned.

1 Like
5

Thanks for the information. OK, that makes sense to me. And yes, the data was only available in one backup set. So something to consider for me the next time.
Nevertheless I will have a look at the retention settings in more detail.

1 Like