Run backup, Delete Files, Run another backup. Previous backup not available. Files seem to be lost

Support
1

Hello, Its been a few years since I’ve used Duplicati. I would like to return to it, but i’m experiencing some ‘show-stopping’ issues, so I thought I’d reach out.

New users are restricted to three screenshots, which is going to make this a little challengeing to show you what i’m talking about.

I have 2.0.7.1_beta_2023-05-25 running on Ubuntu 22.04.

In testing - I am backing up ‘/main/home/’ to ‘/backups’. I have Schedule turned off for testing purposes so i can run manual backups inbetween manipulating data to ensure i can restore no matter what the scenario. I have no advanced options set, and i’m using ‘Smart’ backup retention.

Starting with 4 files: 1.mp3; 2.mp3; file01; file02
I run the first backup. This completes and shows i have 1 version of the backup (as expected).

If i attempt to restore - i can see the files all available.

Next i add 4 more files; 3.mp3; 4.mp3; file03; file04 - So i have 8 files in total.
I run another backup - this shows i have 2 versions of the backup (as expected)

Next, i remove random files, 3.mp3; 4.mp3; file01; file02
I run another backup. This shows me i have 2 versions of the backup.

When i go to restore I only have two options, the FIRST and THIRD backup that was run.

image
image826×231 19.3 KB

The first backup shows the initail files from the first backup available to restore.

The third and most recent backup shows the files that were available at the time of the backup;

image
image1285×630 34 KB

So i have no way of restoring files; 3.mp3; 4.mp3;

====

To further test, if i delete all remaining files from the source and run another backup
Backup shows 2 versions available.

When i go to restore, i can only restore from the FIRST and FOURTH backup;

Below shows files available on FIRST (left) and FOURTH (right) backup

image
image1483×463 47.5 KB

So, essentially, the files added before the SECOND backup are now no longer available to restore.

Please help!
Thank you
-Steve

2

Welcome to the forum @shoodidagen

Version retention is controlled by Options screen 5. What’s yours? I’m wondering if it was:

image

because this (or Custom backup retention) may delete versions-in-the-middle, as you had.
It’s sort of like creating and deleting a file between backups – file winds up not in a backup.

3

There is sometimes a chance to painfully recover deleted files, but not after a compact has run.
I suggest you set no-auto-compact and don’t request a manual one while the investigation runs.
You could look in your job logs to see if it’s too late already, but this all depends on the situation.

4 
1st 14:38       1.mp3; 2.mp3; file01; file02
2nd ?           1.mp3; 2.mp3; file01; file02; 3.mp3; 4.mp3; file03; file04
3rd 14:43       1.mp3; 2.mp3;                               file03; file04 
4th 14:47

They only existed briefly, and only in version that got deleted by retention policy (whatever it is).

These also existed only briefly, although file03 and file04 existed a bit longer than two new .mp3.

The GUI doesn’t say it, but it generally won’t delete a backup immediately after it’s been created.

5

Hello ts678,

Apologies, I did put which retention being used in my original draft, but I must have removed it due to restrictions in amount of screenshots i can upload and forgot to put it in plain text.

To confirm, I’m using ‘Smart Backup Retention’.

As soon as i have time, I’m going to run the same test with ‘Keep All Backups’ to see if its the smart retention that’s doing it.
Thanks
-Steve

6

Hello ts678,

The second backup was roughly 1440hrs (Sorry, it was in screenshots but I had to remove them).
However brief, would it not make sense that a backup would still keep these files? Do files need to exist for ‘x’ time before they are kept?

I plan on re-creating this scenario tonight with a fresh install - I’ll leave 10 minutes between backups, too, to keep track of things a little better.
I’ll take notes of the backup times and show files in the format you’ve shown above.

Thanks,
-Steve

7

is how backup versions work. If a version is deleted, it’s not going to move a file to wrong version.
In the case where every backup version has a different version of the file, similar problem occurs.
You delete the version, and that version of that file is gone. It’s the price of trying to thin a backup.

If you have an actual situation with very short-lived files, you should think hard about the retention.

To see retention in action, you can watch a log-file or live log at information level to get the details.
Raising the level to profiling will show more, but too much unwanted, so you could try a log filter…

EDIT:

Example of Information level log for Custom backup retention of 1D:4h,2W:8h,4W:1W,12M:1M:

2023-12-06 10:05:09 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed
2023-12-06 10:05:09 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 1.00:00:00 / 04:00:00, 14.00:00:00 / 08:00:00, 28.00:00:00 / 7.00:00:00, 365.00:00:00 / 31.00:00:00
2023-12-06 10:05:09 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 12/3/2023 8:45:02 PM, 12/2/2023 2:41:55 PM, 12/1/2023 8:33:00 PM, 11/30/2023 8:34:15 AM, 11/28/2023 9:53:56 AM, 11/26/2023 8:54:54 PM, 11/24/2023 11:09:38 AM, 11/22/2023 1:51:38 PM, 11/21/2023 3:46:27 PM, 11/20/2023 7:58:32 AM, 11/18/2023 7:42:17 PM, 11/18/2023 7:16:00 PM, 11/10/2023 11:46:25 AM, 11/4/2023 7:35:31 AM, 11/3/2023 7:24:34 PM, 10/18/2023 7:50:07 PM, 9/18/2023 9:44:10 AM, 9/11/2023 11:17:25 AM, 8/10/2023 7:40:41 AM, 5/14/2023 1:39:25 PM, 4/10/2023 3:27:54 PM
2023-12-06 10:05:09 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted:
2023-12-06 10:05:09 -05 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: 11/21/2023 3:46:27 PM, 11/20/2023 7:58:32 AM

If you want no limit, you can use U for unlimited interval of backups (or also for unlimited time frame).

8

I think I understand.
Because of the retention policy, I only allow Duplicati to keep so many backups, so the test with Smart Retention (There will remain one backup for each of the last 7 days) will not work if i’m testing backups every few minutes and i’m adding/removing files each time I run one of those backups.

Please do let me know if that is correct?

I really appreciate your time here, it has been really helpful.

Thank you,
-Steve

9

is the main part. It’d delete versions even if you weren’t adding/removing, but it’d raise fewer questions.

1 Like