Purge strange behaviour

Hello
First time here, I am trying to see if Duplicati is the right thing for me
I do not understand something
I want to delete backups of files that I will never need anymore (or were backed up by mistake)
I did a list of the files with the list command line and I saw my backups entries
Then after RTFM , I did a command line Purge of the files
I did a list again, and I do not see them anymore
Tried a FIND command : same… nothing
Looks good !
But then (suspicious as always) I went to do the backup restore menu and I still see the files there
Puzzled , I did a restore of one of the files , and It worked.
So in fact it has never been purged really! It just does not show anymore
I tried some options , like allow-full-removal or putting all versions
No luck

Now can somebody explain ? Is this only a purge in the index of the database or what ?
Is the manual wrong ? ( Purges (removes) files from remote backup data / By default, the matching files are purged in all versions)

Before the pro’s hopefully clarify this, let me chime in as a user who is also just struggling with a possibly related case.
My understanding is that PURGE is used to erase specific files (possibly limited to given backup versions) out of the repository, while DELETE is used to erase alls files of specified backup versions. What did you try to achieve?

Hi
I am tryng to delete specific backup files or specific folder and the proper command seems to be Purge
I do not want to delete specific backup versions (delete command) because this erases all files in the backup set. ( it works as described, I tested it)
And I want to understand why before I run the PURGE , the find or list command gives me the proper results and when I run it after the PURGE it tells me there is no such files while I can see them in the restore and I even can restore them properly
It should not be there anymore (you can’t even list it !)
Looks like it affects only the index in the database not the files themselves

Well I tested again Delete ( you made me think again)
And in fact it is a very similar behaviour as it does not delete the backup
It deletes the entries , but as the size of the backup does not change, it proves that it does not remove the
data.( I tried to take out several gig to make sure)
But this time you can’t know it because it does not show on the restore list of version
So my remote space usage will continue growing for ever (and this is not good with my finance :grinning:

The remote volumes need to be compacted after a purge to free up the space, and even then it is possible Duplicati may decide not to compact some volumes. (There is a threshold that must be reached before compaction will occur.)

So try running the compact command and see if that helps!

Hi
The COMPACT command is automatic after a PURGE ( that is what the book says) and to make sure I ran a PURGE after and it did not change ( but remember that RESTORE shows the file still exists)
On the other hand, COMPACT is not automatic on the DELETE ( at least the book does not say so)
So I ran the COMPACT command as per your suggestion and it saved 5 gig but this is not very clear as I deleted 30gig of backup files ( and I cannot see anymore in the restore list in a find or in a list command ) .
Still digging and the original question remains why can I restore files that LIST and FIND can’t see anymore ( obviously because they are still there)
I have to travel later today , so 'ill continue my investigation at the end of the week
But I smell a fish :laughing:

I believe the previously mentioned threshold also applies in individual remote volumes… In other words, if it can reclaim just a tiny amount of wasted space, downloading, repackaging, and uploading isn’t worthwhile.

The COMPACT command describes the settings. Running compact checks, but only compacts if needed.

This may or may not be what’s behind your small space reclaiming. Theory is worse if backup is very new, because in that case the blocks of the source file are more likely clustered. Entire volumes can be deleted.

Compacting decision can be seen at Information log level in About --> Show log --> Live or in a –log-file

2020-06-26 11:42:06 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)
2020-06-26 11:42:07 -04 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 40.81 MB
2020-06-26 13:42:01 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required
2020-06-26 14:42:11 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there is 25.93% wasted space and the limit is 25%
2020-06-26 14:45:47 -04 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Downloaded 10 file(s) with a total size of 423.49 MB, deleted 20 file(s) with a total size of 423.98 MB, and compacted to 4 file(s) with a size of 74.95 MB, which reduced storage by 16 file(s) and 349.04 MB
2020-06-26 15:41:43 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)
2020-06-26 15:41:44 -04 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 31.72 MB
2020-06-26 16:48:10 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required
2020-06-26 17:42:21 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)
2020-06-26 17:42:23 -04 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 40.94 MB

Simple test works fine here, using GUI Commandline to run The PURGE command on a file from a folder.
File is no longer visible in either the GUI restore tree, or The FIND command run in the GUI Commandline.

If you get a simple test to fail, please detail all the steps. Good issue reports are what helps lead to fixes…