Backend quota is close to being exceeded

No i didn’t deleted anything but even after reading your explanations i don’t understand how to delete my unticked and exclude items :frowning:

Should like enter manually all files i unticked and exclude following the purge command ? It seems hard.

I remember Crashplan when unticking source items : if i’m right it was deleting automaticly after replying “Yes” to the question "Are you sure ? " … or something like that. But maybe i’m wrong, and i know, Duplicati isn’t Crashplan :wink:

The now-discontinued CrashPlan Home used to, IIRC, handle deselection and purge as one operation, giving a big warning you needed to click through to confirm that you were sure. In that sense, it was less hard than the Duplicati way. I don’t know if Duplicati’s method meant to give more control, or was just a simpler first plan. There’s a lot in Duplicati that could be enhanced, but that’s to be expected at its stage.

Whatever you unticked, you probably need to copy into the box in a format similar to what’s there, with paths ending in backslash for a folder and (though you don’t show any on your limited screenshot) files without it. Remove lines you don’t want purged, and always use --dry-run to see if it looks as intended. Copying the list of files and folders before you do that will make it easier to run again as the for-real run.

As for your excluded items, if they were always excluded, then there should be nothing to delete. If you added excludes over time (and keep all versions forever) then you might have some copies before the exclude that will take up space forever. You possibly could purge those with a separate purge using the --include to include them in the purge, which is the opposite of using --exclude to exclude from backup.

But I almost never purge, so please use --dry-run to check. Purge is permanent, just like on CrashPlan.

Ok.

Maybe i can retreive on my notes what i unticked, but i don’t understand how you could see what i unticked on my screenshot : once i unticked items and save my backup config, there isn’t indication on what has been untickicked ?!

Maybe you mean the entire list in “Commandline arguments” which is reduced and match all my source items ? For the rest, my screenshot isn’t limited : you can display all of it by clicking on it.

Some exclude filters are old but some are new (since this discussion).

Yes, i will try with --dry-run :slight_smile: I keep you informed.

Thanks

is a non-specific way of assuming you unticked items (as you said), and saying what to do as next step.
Unfortunately

It does not save your entire history of what you did to the backup configuration. CrashPlan doesn’t either.
If you have a previous export of the configuration (which you should for safety), you can look at it either in native form which you can open in notepad or whatever and try to interpret, or import for a look, but don’t save it unchanged or you may have two identical backups clobbering each other in the same destination.

Another way of figuring out what you previously backed up is to start down restore just to inspect the tree.

The FIND command can also be used to show the files in the backup, but it’s a command-line tool again.

Yes. All visible ones ended in backslash.

That gives an unreadably tall and narrow view, which I did download and open in Paint 3D to see excludes.

For example, possibly it’s the reason you’re so carefully picking what to back up. I know I’m a bit careful of cloud storage usage because it’s metered, but the same vendor (Backblaze) has an unlimited backup too, intended to simplify things so people don’t pick-and-choose and possibly omit important things by mistake. One drawback is they only keep deleted data for 30 days. Another grab-everything option might be to get a drive (maybe even your current one if you upgrade) for use as an occasional image backup to get another. Macrium Reflect Free can do a full image and then you can maintain it with smaller differentials if you wish.

Good luck!

Maybe it isn’t the good terms (i’m french). What is the good expression for that ? Unchecked ?

Crashplan didn’t either but it wasn’t needed if i replied “Yes” to the question about “delete files in backup”.

All these tips are useful, thanks :slight_smile:

Not for me : if i click on the screenshot it open a first tall and narrow view, but after a second click it is perfectly visible. Maybe it depends of browser used…

Yes i know, it’s why i didn’t choosed BB but iDrive instead.

I already use Macrium to do full image and differentials.

Duplicati isn’t my only one backup solution, i have several ones and i am well protected.

My present discussion is for avoid, if possible, to spend money for a new disk and also to learn how the program works.

Because i have others backup solutions, it isn’t a problem if i lose my program data files history on Duplicati, but I did not think it could be so hard to free up space after “unchecking”? :wink: or excluding items. I thought to a one action like Crashplan did.

I don’t know how it will be hard to add this feature but it could be great : Purge “all data which aren’t selected in data source and which are excluded by filters”.

Thanks again for your big help :slight_smile:

But this isn’t what custom retention does. Custom retention trims out unneeded versions between old versions - but (assuming you set it up correctly) will always keep SOME versions from even the oldest timeframes (see the config suggestions by others here… most of the ones I’ve seen in this thread keep some old version almost indefinitely). So don’t look at it as deleting data past a certain scope, just think of it as strategic thinning out of old versions.

Hi :slight_smile:

Yes i understand. I will surely try a custom retention setting soon :slight_smile:

For now i’m trying to purge my unselected and exclude items.

After some errors messages and tests i think i managed to start a purge from the command Duplicati “GUI” with that (in real, i put more folder paths in commandline arguments) :

The process started ! And you know what ? I forgot to add the --dry-run advanced option !!
Nevermind (i have others backups solutions), i left the process do its job… and it took several days to finish !

This is what i had displayed in GUI :

        Execution d'une ligne de commnde
        Terminé !
        Listing remote folder ...
        Backend quota is close to being exceeded: Using 1.42 TB of 1.79 TB (136.23 GB available)
        Uploading file (47.65 MB) ...
        Deleting file duplicati-20180613T142533Z.dlist.zip.aes ...
        Uploading file (47.66 MB) ...
        Deleting file duplicati-20180614T014858Z.dlist.zip.aes ...
        Uploading file (47.75 MB) ...
        Deleting file duplicati-20180614T230001Z.dlist.zip.aes ...
        Uploading file (47.77 MB) ...
        Deleting file duplicati-20180615T230003Z.dlist.zip.aes ...
        Uploading file (47.75 MB) ...
` Return code: 0`

I deleted a lot of lines but it was the same : “Deleting file…”

I received also this report by mail :

MainOperation: PurgeFiles
RemovedFileCount: 7605
RemovedFileSize: 0
RewrittenFileLists: 448
ParsedResult: Warning
Version: 2.0.4.23 (2.0.4.23_beta_2019-07-14)
EndTime: 24/09/2019 03:18:32 (1569287912)
BeginTime: 19/09/2019 09:34:25 (1568878465)
Duration: 4.17:44:06.8767000
Log data:
2019-09-19 09:50:22 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaNear]: Backend quota is close to being exceeded: Using 1.42 TB of 1.79 TB (136.23 GB available)

And as the last line says : “Backend quota is close to being exceeded”, always :frowning:

I check my storage space : no more free space :

I edit my backup config and delete items (paths) i purged from my data source selection, and i also delete all my filters “exclude”, then i start a new backup.

I receive this mail report :
DeletedFiles: 70569
DeletedFolders: 8159
ModifiedFiles: 137
ExaminedFiles: 310465
OpenedFiles: 313
AddedFiles: 176
SizeOfModifiedFiles: 248311028
SizeOfAddedFiles: 45042321612
SizeOfExaminedFiles: 1061428117014
SizeOfOpenedFiles: 45318912021
NotProcessedFiles: 0
AddedFolders: 20
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
ParsedResult: Warning
Version: 2.0.4.23 (2.0.4.23_beta_2019-07-14)
EndTime: 24/09/2019 10:15:05 (1569312905)
BeginTime: 24/09/2019 07:45:51 (1569303951)
Duration: 02:29:14.2280000
Log data:
2019-09-24 09:35:16 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaNear]: Backend quota is close to being exceeded: Using 1.42 TB of 1.79 TB (136.23 GB available)

The beginning sounded good (deleted files and folders) but not the end : “Backend quota is close to being exceeded”…

I check my storage space : no more free space.

I try “Compact now” from GUI : no more free space…

Am i missing something ?

1 Like

As compact repackages partially empty volumes into new ones, it looks like it doesn’t delete the old ones until the repackaging finishes OK. This makes sense to prevent loss of data, but means free space gets lower before it gets higher, i.e. a case of “Living with an almost-out-of-space condition can get awkward.” which wouldn’t be noticed if things didn’t start out so tight. Assuming this is the problem, a way out might be to do compact gently at first, eventually returning to the default values, or maybe even going beyond…

The COMPACT command

documents three options, but –small-file-size and –small-file-max-count are probably not relevant here if compact was not disabled previously with –no-auto-compact. The option to ramp down on is –threshold, starting with a high wasted space percent, then slowly dropping. But first you need some space, and I’m not familiar with the best way to recover. Maybe backup will do it. If not, try repair, but it can be surprising.

Hi,

You mean that my 135 GB free aren’t enought to make the compact doable ?

Maybe i found another explanation, or at least, there is something weird :frowning:

I tried, just to see, to use the “restore” feature.
If i choose the last version, my “purged” data isn’t displayed, that’s normal (i unchecked items from Source data config).
But, if i choose an older version, i can see all my purged (i thought purged) data; for instance i can list and recover all my Dropbox data !

It is like i made a “DryRun” purge ?! But as i said : i forgot to add the advanced option --dry-run .

Did the “–log-file-log-level=DryRun” advanced option made a DryRun Purge ?

Thanks

Which is it? No more or 135 GB. Did Duplicati fill up, then release? If so, great – less manual work to do making a small amount, but you’d still need to test that slow gradual compact to see whether it’s doable.

It’s not documented as that, and if it truly did --dry-run then nothing should have gone away. Possibly you didn’t actually purge anything significant. I have a feeling (having just done a purge to look into a bug from 2.0.4.29 and 2.0.4.30 canary (I hope you’re not on either – any beta is fine)) that the right way to purge an entire folder is to use --include, e.g. with wildcards, e.g. * means any character, e.g. filenames in a folder. Putting double quotes around the whole wildcard pattern is probably safest, e.g. --include="<whatever>*"

If your restore shows an all-version deletion of a file that you specified individually, then that fits this theory.

Hi :slight_smile:

I think it is a translation problem : by “no more free space” i mean “no freed up space” ? : my 135 GB free space stayed the same : the process didn’t increased it.

Not sure to understand. During the 3-4 days of purge, i checked sometimes my space with windows explorer : nothing was changing.

If i did a purge, it is normal to have then to do a compact to free up space ?

It’s my feeling : nothing seems to have been deleted.

I think i should win at least some GB with the purge. My Dropbox is about 10 GB; my evernote too, both with a lot of differents versions.

I’m on last beta.

I didn’t use any wildcard or quotes, this is how i start the purge (from GUI) :

So, i should do the same but with nothing in “Commandline arguments” and each folder i want to delete in “Advanced options” with include filter ? Like this ?

--include="D:\Dropbox\*"

or i can just use again the “Commandline arguments” with my folders like this ? :

“D:\Dropbox*”

By “individually”, you mean “only a file” specified for items to purge ? Non, i only choosed “folders” to purge. But all folders and containing files are listed in th restore backup gui.

Thanks

Hi :slight_smile:

I think i’m close to succefully purge my files.

With your help, I managed to purge one folder from command line GUI.

Then, i added same include filters for purge several folders in one time, but it seems that only the last line is considerated.

For instance, with these lines :

--include="G:\OUTILS\Linux\Lubuntu\*"
--include="G:\OUTILS\Linux\Raspberry\*"
--dry-run

Only the “Raspeberry” folder is purged (or displayed in DryRun log).

Can you help me with that ?

Thanks

Works fine here on 2.0.4.30 --dry-run. The following is what I put in the Commandline arguments box. Here, I don’t think you need quoting (that’s for true command line use) but you do need separate lines.
Also make sure you don’t have any inappropriate filter --include or --exclude left over from original job.

--include=C:\backup source\A.*
--include=C:\backup source\B.*
--dry-run
--console-log-level=DryRun
Running commandline entry
Finished!

            
  Listing remote folder ...
[Dryrun]:   Purging file C:\backup source\B.txt (1 bytes)
[Dryrun]: Would write files to remote storage
[Dryrun]: Would upload file duplicati-20190929T001942Z.dlist.zip (233 bytes) and delete file duplicati-20190929T001941Z.dlist.zip, removing 1 files
[Dryrun]:   Purging file C:\backup source\B.txt (1 bytes)
[Dryrun]:   Purging file C:\backup source\A.txt (1 bytes)
[Dryrun]: Would write files to remote storage
[Dryrun]: Would upload file duplicati-20191011T111403Z.dlist.zip (233 bytes) and delete file duplicati-20191011T111402Z.dlist.zip, removing 2 files
Return code: 0

This backup has two versions currently, the first with B.txt and C.txt. The newer version also has A.txt.

Hi :slight_smile:

I don’t put anything in Commandline arguments (in green on the screenshot below). When you told me to use --include filter, i thought i had to put them in “Advanced options” box. It is maybe the cause of my problem.

This is my modified command line GUI after deleting the content in commandline arguments box and all “exclude filter’s” in advanced options. My modifications are in red :

So i have to put include filters in Commandlinearguments box ? not in advanced options ? I see you also put --dry-run and --console-log-level-DryRun in the same box… I don’t understand clearly the differences between the 2 boxs; i thought that all “–…” was options to put in the second…

Thanks

I doubt it. There are about three ways to do entry, and they should be equivalent AFAIK. The format of the Commandline screen is oriented around The BACKUP command which has command line syntax:

Duplicati.CommandLine.exe backup <storage-URL> "<source-path>" [<options>]

and the source paths come from the Commandline arguments box. You need to at least clear it when source paths don’t apply, but it’s easy to just keep typing, so options there just go before other options.
Your way of Advanced options with Edit as text should be the same. You can test both if you like.

Even though my tests didn’t seem to mind quotes, I know there has been a report of quotes hurting, so maybe you could take the “Enter one argument per line without quotes” shown, and use it for your edit.

True commandline (as opposed to GUI’s Commandline) can need quotes, e.g. your --send-mail-subject with its embedded spaces would almost certainly fail on Linux (due to shell) unless it was inside quotes. Windows is different, but if you Export As Command-line you see when Duplicati chose to quote values.

Hi,

Thanks a lot for your explanations.

I managed to purge multi folders at one time using Commandline arguments and wildcard at the end of each path :

It worked on my new backup job (very recent with only 2 versions).

So i’m actually trying a purge on my main backup with the same way, but it is very long (already almost 3 days and i’m not at the half of the process). I’m confiant because i can already see that my storage place is winning free place :slight_smile:

I keep you informed.

Bye

Hi :slight_smile:

The Purge worked ! I have freed up 30 GB :slight_smile:

It is good but not enough, so i’m now trying to set a custom retention to freed up more space.

I choosed : 2M:1D, 6M:1W, 1Y:1M, 2Y:3M, 5Y:6M, U:1Y
If i’m right it means :
Keep a daily backup for the last 2 months (so about 60 versions max)
then keep a weekly backup for the last 6 months (so about 24 version max)
then keep a monthly backup for the last year (so about 12 versions max)
then keep a backup each 3 months for the last 2 years (about 8 versions max)
then keep a backup each 6 months for the last 5 years (about 10 versions max)
then a yearly backup forever

Am i right ?

I tried to start the backup and i had an error message :

I changed U for 99Y : backup is starting… But how can use the “U” ?

Thanks

Yes, assuming “max” means you know that time frames nest, so 5Y only starts after the 3Y one ends.

Remove extra spaces. This is supposed to be a comma-separated list, but the number parser is more forgiving of leading spaces apparently. The “U” detector runs first, sees a space which is not a U, so it tries to process it as a number which doesn’t work either. The translation of the message seems to be

msgid "Unparsed data: {0}"
msgstr "Données non analysées: {0}"

Hi :slight_smile:

Ok, i hadn’t understood that.

Ok, but in the GUI there is an example with space :

So, i changed my code to 2M:1D,4M:1W,6M:1M,1Y:3M,4Y:6M,U:1Y

If i’m right, now it means :

Keep a daily backup for the last 2 months (so about 60 versions max)
then keep a weekly backup for previous 4 months (so about 16 version max)
then keep a monthly backup for previous 6 months (so about 6 versions max)
then keep a backup each 3 months for previous year (about 4 versions max)
then keep a backup each 6 months for previous 4 years (about 8 versions max)
then before that, a yearly backup forever

All good ? :slight_smile: (backup started…)

The translator (probably at Transifex) for French changed both comma-separated strings to add spaces. While it’s a better style for human writing, computers are more literal, and probably throughout the code. Doing a search for both strings with and without extra space found all other languages just use commas.

I’m not sure exactly how translations get done, but maybe someone else will know how to fix the error… Writing it up in a Duplicati GitHub issue would at least record a request. Support requests aren’t tracked.

Alternatively you could ask that leading spaces in front of the U be ignored, but in general I think comma separated means just comma. The fact that the number field took leading spaces is a bonus accident…