Backend quota exceeded (I know it's been covered, but...)

Support
1

Hi folks,

I know this topic has been covered many times but I’ve read through every instance without success. I was previously using an attached drive, with no problems. However, for obvious reasons, I wanted my backups to be remote - but I’m not in a position to fork out for that - so I came up with the solution of backing up via usb cable to a folder in my android phone (which is permanently with me and has plenty of space). Now, however, although every job is listed as successfully completed it results in the following error:

[Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 103.17 GB of 0 bytes (0 bytes available)

There is more than 300GB of free space on the phone, and I have it allocated as a ‘local folder or drive’ which, while connected, it is. I’ve checked my logs for errors, and the backend statistics show no errors other than “ReportedQuotaError”: true, and I’m also baffled by the “total” and “free” quota space - please see below:

“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: true,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,

OS is Linux Mint. Any help greatly appreciated.
pmd

2

For a disk backend, the quota is the available space reported by the operating system.
So if you have mounted your backend on, say, /backup , what says df /backup ?

3

Hi gpatel,

Here’s what I get:

Filesystem 1K-blocks Used Available Use% Mounted on
gvfsd-fuse 990119916 175546404 814573512 18%

Note that it’s 500GB internal storage (where I backup to) and 500GB SD Card. The fre space free internal storage is actually 334GB.

Hope this is of use.

Regards,
pmd

4

Do you run Duplicati as a user ? Fuse mounts are a bit tricky.

5

Crikey, gpatel, that had never even occurred to me.

It’s my personal PC and I boot login as administrator but I may be just a user after that (although I can access anything with root privileges with the appropriate password).

I’ve finished for today (long story) but will check tomorrow and get back to you. Should I be accessing it as a user or not? And how do I check / change it if necessary?

Really appreciate your time.
pmd

6

I can reproduce this on linux. Android uses MTP to transfer files, so it is mounted as a fuse mount.
This is a problem with Mono. System.IO.DriveInfo.GetDrives() finds the folder above the mount point (/run/user/1000/gvfs/), and that returns 0 for the sizes (also with df):

/: AvailableFreeSpace 6145687552, DriveFormat ext, DriveType Fixed, IsReady True, RootDirectory /, TotalFreeSpace 7420743680, TotalSize 25859465216, VolumeLabel /
/run/user/1000/gvfs: AvailableFreeSpace 0, DriveFormat fuse, DriveType Ram, IsReady True, RootDirectory /run/user/1000/gvfs, TotalFreeSpace 0, TotalSize 0, VolumeLabel /run/user/1000/gvfs

The correct mount point would be /run/user/1000/gvfs/mtp:host=XXX/. df -a also does not list this mount, but specifying the path directly works. I tried to put the correct path in the DriveInfo constructor, but that also falls back to the parent directory. I am not sure what we can do about this. Maybe add a check that total size is zero and disable quota warnings?
It seems that mono uses statvfs to get the statistics, so it may even be something in that library function.

7

or it may be unrelated to mono, but to the fact that MTP is not an USB file system. See for example this blog post:

the relevant quote being:

jmtpfs does not support reporting blocks, because MTP hides fs details by design. It always sets st_blocks=0, which makes du unable to report disk usage in blocks.

so disabling quota warnings if reported size is zero seems just as well indeed.

8

Automatically ignoring values that seem kind of insane (e.g. 0) is nice, but providing a manual option to turn off quota might be the best way to bypass the mono bug. Using df or doing system calls is messy.

9

All this help is really appreciated, folks, but I confess you are already exceeding my Linux knowledge.

From what I think I understand of your respective responses, mono / statvfs is reporting an error despite the fact that there is actually nothing wrong with my backups. Do you all agree that is the case because, if so, I’m happy to just carry on receiving that error after each backup, and ignoring it.

10

All this help is really appreciated, folks, but I confess you are already exceeding my Linux knowledge.

From what I think I understand of your respective responses, mono / statvfs is reporting an error despite the fact that there is actually nothing wrong with my backups. Do you all agree that is the case because, if so, I’m happy to just carry on receiving that error after each backup, and ignoring it.

11

All this help is really appreciated, @gpatel-fr @ts678 @Jojo-1000, but I confess you are already exceeding my Linux knowledge.

From what I think I understand of your respective responses, mono / statvfs is reporting an error despite the fact that there is actually nothing wrong with my backups. Do you all agree that is the case because, if so, I’m happy to just carry on receiving that error after each backup, and ignoring it.

12

Yes that’s exactly what I think, discussion was more about finding a way to work around this limitation in a future release.

13

Thanks @gpatel-fr - @Jojo-1000 and @ts678, are you in agreement?
pmd

15

As far as I can tell, this is only an error message and it does not stop any of the backup processes. In my tests the files were copied properly (you should also see that files are there in your target folder).

I remember there was an issue about showing the size of the wrong mounting point. Maybe we could disable quota checks completely if --quota-warning-threshold=-1, and also if the total free space is 0 (because that seems to be the signal by mono that it can’t get the stats). I think we can reuse the flag because if you disable the error, you probably don’t want to get a warning either (unless you find this too confusing).

16

OK, I found some numbers, but they look odd. Total’s nearly a thousand GB. Is it a combined value?
On Ubuntu 22.04, df (and everything) gives me only 0s for /run/user/1000/gvfs. I don’t have mtp.

Expected behavior of a setup like this is unknown to me. The available here looks about 500 GB high, meaning we can’t trust it even if we could get it, if it’s actually true that 334GB is the available amount.

  --quota-size (Size): A reported maximum storage
    This value can be used to set a known upper limit on the amount of space a
    backend has. If the backend reports the size itself, this value is
    ignored

The last sentence IMO has too much trust in the backend and subsequent reporting code, especially if destination is Local folder or drive simulated by who-knows-what-mechanism. There’s an old issue for

and if this were allowed, perhaps it would provide a general way out of a predicament like the current.

Backup Quota Warning Error #4970 has some comment but is waiting for information from originator.

Backend quota has been exceeded after apparently successful backup has earlier technical analysis.

There might be other short-term solutions, e.g. rather than use media transfer protocol, use a network.
There appear to be FTP (somewhat insecure unless encrypted) and SSH servers you can investigate.

17

So I now have these ideas:

  1. --quota-size overrides reported value from backend.
    I don’t even think this option works currently, it only sets AssignedQuotaSpace which is never used to create a warning or error.
  2. --quota-size-override to do this with a new option instead
  3. --quota-size should probably be used instead to specify the maximum size of all Duplicati-created files, and create a warning when the threshold is met (in addition to the reported backend quota). I think if this is exceeded, we still complete backups and only show an error. That might fill up the space, but at least it won’t suddenly stop making backups.
  4. --quota-warning-threshold=-1 to disable quota warnings and errors
  5. --quota-disable to do this with a new option instead
    Here we could also have --quota-disable=all and --quota-disable=reported to still have the manual quota size and ignore the reported quota which might be wrong. Or maybe only disable the reported quota and have the user remove --quota-size to disable that as well.

I think I would prefer to do 3 and 5, so that --quota-size does something meaningful in all cases (although not compatible with the quoted description, as I said currently it does nothing) and there is also the option to disable the quotas.
Manually specifying the full backend size (as opposed to only Duplicati files) will not work so easily, because if the reporting does not work we can’t know how much of that is used by other files.
@gpatel-fr @ts678 do you agree with me? Then I am going to implement this.

18

It looks like it never worked in Duplicati 2. I traced help text source to a comment as of Duplicati 1.3.4.

Good point. An equivalent definition when destination has no quota information is to limit Duplicati use, following similar semantics (which I might have wrong, so read the code) for warning, and error at limit.

For the needs of the issue I linked, one can do the same thing by giving less than the full backend size.

  --quota-warning-threshold (Integer): Threshold for warning about low quota
    Sets a threshold for when to warn about the backend quota being nearly
    exceeded. It is given as a percentage, and a warning is generated if the
    amount of available quota is less than this percentage of the total
    backup size. If the backend does not report the quota information, this
    value will be ignored
    * default value: 10

Duplicati could do math to figure out the above, maybe something using Complete log names

      "UnknownFileSize": 0,
      "KnownFileSize": 4002,
      "TotalQuotaSpace": 999618043904,
      "FreeQuotaSpace": 317370253312,
      "AssignedQuotaSpace": -1,

In manual override mode, TotalQuotaSpace comes from --quota-size which is taken when given.

TotalQuotaSpace = AssignedQuotaSpace --quota-size
“total backup size” = KnownFileSize + UnknownFileSize
FreeQuotaSpace = TotalQuotaSpace - “total backup size”

I prefer to stick near current behavior (fixed as necessary), and limit the number of interacting options.
There’s no manual section on quotas, but I guess having options be alphabetical helps to group them.
Sort of the “as simple as possible but not simpler” plan. I suppose one could go study reported issues.

I think what I detailed is 3? I’m not sure 5 is needed, especially in two flavors. Starting to get confusing. Disabling quota should also not be encouraged by offering an option. Again, one could research some. Who’s asking for it, and why? If it’s available, then used, how messy does it get when space exhausts?

19

I’d say that any complicated fix for a bug that is not a Duplicati bug in the first place seems a strange idea to me in the first place. If I was bothering for that I’d go for a disabling flag with a one-liner change.

20

Hi @ts678 . Remember, that’s the whole phone (500 GB internal - where I backup to - and 500 GB SD card). I did try to df just the internal storage but it reported no such file or directory. However, all the backup files are in the internal folder where I directed them. I’m now assuming that the error report is false and am ignoring it and continuing with that method - unless you think that’s a bad idea?
pmd
@Jojo-1000 @gpatel-fr

21

I don’t think this is too complicated, also since --quota-size does not work at all right now.
My suggestion is:

  • Make --quota-size work in addition to the reported backend quota, but only for Duplicati files. This way it could also be used to monitor the backup size itself.
  • Add a flag to disable the reported quota in case it is wrong
  • Disable quota for local files if total size is 0

The help could be as simple as

--quota-size (Size): Limit storage use
Set a limit to the total amount of storage used on the backend. This is in addition to the full backend quota, if available.
Note: Backups will continue past the quota. This only creates warnings and error messages.

--quota-disable: Disable backend quota
Disable the quota reported by the backend. --quota-size can still be used to set a manual quota.

However, @ts678 thinks that a new option to disable is unnecessary and instead --quota-size should replace the reported quota if present.

Here is a scenario:

  • I back up to a drive (1TB) that is 50% full with video games
  • I only want Duplicati to use 200GB because I want to have space left. I could calculate this with --quota-warning-threshold, but that is complicated so I set --quota-size instead.
  • I install another 400 GB of games on the drive. Duplicati now only has 100GB to work with, but a quota of 200GB.
  • If the quota size flag replaces the reported quota, it will try to completely fill up the drive without a warning and fail at some point once it is full.

This is why I want the flag to be in addition, rather than replace the reported quota. The flag to disable quota is then only needed if there is some bug with it (for example it reports the wrong mount point).