Backup to Onedrive fails with error

optismiranda · December 9, 2021, 4:46pm

Hello,

I am using Duplicati 2.0.6.3_beta_2021-06-17 on Windows 10 Pro to make backups to OneDrive.

In some cases I receive the following errors:
LimitedErrors: [
2021-12-09 12:44:59 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 61,09 GB of 1,00 TB (0 bytes available) ] Log data:
2021-12-09 12:44:59 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 61,09 GB of 1,00 TB (0 bytes available)

As you can see from the error, I am just using 61 Gb of 1 TB

What can I do to solve this problem?

Thanks and Regards,

Sebastian

drwtsn32 · December 9, 2021, 4:57pm

This thread is similar: Backend quota has been exceeded

Sounds like in that case, the OP had their destination set to “local folder or drive” and had it pointed to the local OneDrive folder. Is that how you have it set up? If so it should work better to set the back end to “Microsoft OneDrive v2”.

optismiranda · December 9, 2021, 6:21pm

@drwtsn32 Thanks for your reply, in fact I have it configured to the “Microsoft OneDrive v2” and have tested the connection with success results and it still fails.

ts678 · December 9, 2021, 10:31pm

Welcome to the forum @optismiranda

That might be a misinterpretation. This backup is using 61 GB, but there could be other users of space.

So it’s intermittent? Please see a job log Complete log section. Under BackendStatistics, please show

(for example – what’s yours?)

      "TotalQuotaSpace": 26843545600,
      "FreeQuotaSpace": 154040560,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,

If your runs that give errors make a log (they might not), you can try comparing those to ones that run OK.
Does the FreeQuotaSpace value look close to the one given by OneDrive (maybe lower left on the page)?

Is the 1 TB value what you expect? If so, it would seem that OneDrive is successfully returning that value.
The Total and Free are (I think) returned together, so I would be surprised if 0 free is due to a query failure.

While you’re in Complete log you might as well look at some other numbers.

KnownFileSize should be your usage, so roughly 61 GB.
RetryAttempts should ideally be 0, or you might have network issues that are playing a part in this issue.

optismiranda · December 10, 2021, 12:33am

Hello @ts678,

Thanks for your reply, I want to clarify some things that could have taken you into confuson

I have several users using Duplicati pointing to OneDrive v2 option. Each user has 1 TB of space assigned

When I said “some cases” I meant some users, when a user gets Error as result and the reason is “[Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]”, that same users always receives this error. I solved it by deleting all backup Jobs and starting all over again which I think is a very dangerous practice.

I will include the complete log tomorrow as of today the user has gone for the day.

Thanks and Very Best Regards,

Sebastian Miranda

ts678 · December 10, 2021, 12:58am

Do you delete remote files along with the jobs? That would say how much you removed. Small example:

The other way to start all over again uses other tools that may or may not show how much was deleted.

Are there administrative tools where you can see quota usage? If so, any idea how it was before delete?

optismiranda · December 27, 2021, 7:47pm

Hello ts678,

I deleted a backup job that has been failing deliting everything, local and remot files. Then I reconfigured it and after about an hour or two I received the following message:

[Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 0 bytes of 1,00 TB (0 bytes available) ] Log data:
2021-12-27 14:44:57 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 0 bytes of 1,00 TB (0 bytes available)

As you can see now the system is using 0 from 1 TB and failing before it was 60,9 GB out of 1 TB What can I do in order to have this backup done?

Thanks and Greetings,

Sebastian

ts678 · December 28, 2021, 2:42am

No answer yet.

Your situation may be unique, and I’m still asking for basic information such as space usage.
I am not a OneDrive administrator, but doesn’t Microsoft provide anything to users or to you?
My OneDrive Personal GUI matches Duplicati’s values seen in BackendStatistics quite well:

    "TotalQuotaSpace": 26843545600, (exactly 25 GiB)
    "FreeQuotaSpace": 519893060,    (1.9% of above) (also 495.8 MiB)

If you’re not able to get Backup to complete, possibly you don’t have a Complete log to cite.
Try using the Verify files button instead. If you get a log, please show me the numbers.
If you or any user has a Microsoft web GUI view of the area, please show me the numbers.

This is Duplicati’s. What about others? The puzzling thing is where the hours of backup went.
Are there files present at the destination? If so, either OneDrive or Duplicati is getting bad info.
It must not be failing entirely because it was saying 60.9 GB before, but does 0 match actual?

optismiranda · December 28, 2021, 3:47pm

Greetings ts678,

I have a new case regarding this same problem.

Yesterday I deleted the backup of one of the affected notebook and tried configuring the back up again with same results.

Today, I configured a new backup job in another notebook with the same problem/error and received the same error.

In both cases notebooks are running release Duplicati - 2.0.6.3_beta_2021-06-17

They both are running Windows 10 and both of them have 1 TB quota in One Drive.

As you can see they both have lots of space available.

User1:

User2:

I configured a new job to an external HD to test if it works, on one of the two notebooks.

I’ll post the results when it has finished.

While I was configuring this job I executed a File Verification and received this error:

. Errors 1

2021-12-28 12:30:14 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 60,65 GB of 1,00 TB (0 bytes available)

Where can I get these values?

Regards,

Sebastian

ts678 · December 29, 2021, 12:48am

Means

On the expanded menu for a backup job, click Show log
Expand the operation of choice to see the log it produced
Click Complete log to switch from friendly view to detail
Look for BackendStatistics, then look for values below

Now that we have some usage numbers, you can also look at these to see if they resemble your values.

    "KnownFileCount": 82,
    "KnownFileSize": 409893621,

It will be interesting to compare quota values. The error message definitely disagrees with your numbers.
I suppose you could also look at some systems not having the failure to see how their values are looking.

optismiranda · December 29, 2021, 1:29pm

Good day ts678,

I was able to get some numbers from one of the notebooks with the problem.

As far as I can see Duplicati reads the FreeQuotaSpace from somewhere with a wrong value as this user has lots of space available

Has used 3.14% out of 1 TB

These are the numbers.

“BackendStatistics”: {

“RemoteCalls”: 9,

“BytesUploaded”: 1706,

“BytesDownloaded”: 56939911,

“FilesUploaded”: 2,

“FilesDownloaded”: 3,

“FilesDeleted”: 2,

“FoldersCreated”: 0,

“RetryAttempts”: 0,

“UnknownFileSize”: 0,

“UnknownFileCount”: 0,

“KnownFileCount”: 1254,

“KnownFileSize”: 30989044542,

“LastBackupDate”: “2021-12-28T10:08:15-03:00”,

“BackupListCount”: 30,

“TotalQuotaSpace”: 1099511627776,

“FreeQuotaSpace”: 0,

“AssignedQuotaSpace”: -1,

“ReportedQuotaError”: true,

“ReportedQuotaWarning”: false,

“MainOperation”: “Backup”,

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “0001-01-01T00:00:00”,

“BeginTime”: “2021-12-28T23:00:00.0138307Z”,

“Duration”: “00:00:00”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null

}

},

“VacuumResults”: null,

“DeleteResults”: {

“DeletedSetsActualLength”: 0,

“DeletedSets”: ,

“Dryrun”: false,

“MainOperation”: “Delete”,

“CompactResults”: {

“DeletedFileCount”: 2,

“DownloadedFileCount”: 0,

“UploadedFileCount”: 0,

“DeletedFileSize”: 1706,

“DownloadedFileSize”: 0,

“UploadedFileSize”: 0,

“Dryrun”: false,

“VacuumResults”: null,

“MainOperation”: “Compact”,

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “2021-12-28T23:01:34.5242792Z”,

“BeginTime”: “2021-12-28T23:01:26.1190246Z”,

“Duration”: “00:00:08.4052546”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null,

“BackendStatistics”: {

“RemoteCalls”: 9,

“BytesUploaded”: 1706,

“BytesDownloaded”: 56939911,

“FilesUploaded”: 2,

“FilesDownloaded”: 3,

“FilesDeleted”: 2,

“FoldersCreated”: 0,

“RetryAttempts”: 0,

“UnknownFileSize”: 0,

“UnknownFileCount”: 0,

“KnownFileCount”: 1254,

“KnownFileSize”: 30989044542,

“LastBackupDate”: “2021-12-28T10:08:15-03:00”,

“BackupListCount”: 30,

“TotalQuotaSpace”: 1099511627776,

“FreeQuotaSpace”: 0,

“AssignedQuotaSpace”: -1,

“ReportedQuotaError”: true,

“ReportedQuotaWarning”: false,

“MainOperation”: “Backup”,

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “0001-01-01T00:00:00”,

“BeginTime”: “2021-12-28T23:00:00.0138307Z”,

“Duration”: “00:00:00”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null

}

},

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “2021-12-28T23:01:34.5242792Z”,

“BeginTime”: “2021-12-28T23:01:23.3602989Z”,

“Duration”: “00:00:11.1639803”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null,

“BackendStatistics”: {

“RemoteCalls”: 9,

“BytesUploaded”: 1706,

“BytesDownloaded”: 56939911,

“FilesUploaded”: 2,

“FilesDownloaded”: 3,

“FilesDeleted”: 2,

“FoldersCreated”: 0,

“RetryAttempts”: 0,

“UnknownFileSize”: 0,

“UnknownFileCount”: 0,

“KnownFileCount”: 1254,

“KnownFileSize”: 30989044542,

“LastBackupDate”: “2021-12-28T10:08:15-03:00”,

“BackupListCount”: 30,

“TotalQuotaSpace”: 1099511627776,

“FreeQuotaSpace”: 0,

“AssignedQuotaSpace”: -1,

“ReportedQuotaError”: true,

“ReportedQuotaWarning”: false,

“MainOperation”: “Backup”,

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “0001-01-01T00:00:00”,

“BeginTime”: “2021-12-28T23:00:00.0138307Z”,

“Duration”: “00:00:00”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null

I ran Verify Files and got

2021-12-29 10:22:44 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 28,45 GB of 1,00 TB (0 bytes available)

The complete Log is the following:

{

“MainOperation”: “Test”,

“VerificationsActualLength”: 3,

“Verifications”: [

{

“Key”: “duplicati-20211228T131456Z.dlist.zip.aes”,

“Value”:

},

{

“Key”: “duplicati-i248b3596656c4efeaf11bf6b96c0a934.dindex.zip.aes”,

“Value”:

},

{

“Key”: “duplicati-b1f3e24bc0f6949449a6233e68136ad82.dblock.zip.aes”,

“Value”:

}

],

“ParsedResult”: “Error”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “2021-12-29T13:22:52.305113Z”,

“BeginTime”: “2021-12-29T13:22:21.9948411Z”,

“Duration”: “00:00:30.3102719”,

“MessagesActualLength”: 9,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 1,

“Messages”: [

“2021-12-29 10:22:22 -03 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started”,

“2021-12-29 10:22:23 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,

“2021-12-29 10:22:39 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (1,14 KB)”,

“2021-12-29 10:22:44 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20211228T131456Z.dlist.zip.aes (4,30 MB)”,

“2021-12-29 10:22:46 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20211228T131456Z.dlist.zip.aes (4,30 MB)”,

“2021-12-29 10:22:46 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i248b3596656c4efeaf11bf6b96c0a934.dindex.zip.aes (17,95 KB)”,

“2021-12-29 10:22:46 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i248b3596656c4efeaf11bf6b96c0a934.dindex.zip.aes (17,95 KB)”,

“2021-12-29 10:22:48 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b1f3e24bc0f6949449a6233e68136ad82.dblock.zip.aes (49,99 MB)”,

“2021-12-29 10:22:52 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b1f3e24bc0f6949449a6233e68136ad82.dblock.zip.aes (49,99 MB)”

],

“Warnings”: ,

“Errors”: [

“2021-12-29 10:22:44 -03 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-BackendQuotaExceeded]: Backend quota has been exceeded: Using 28,45 GB of 1,00 TB (0 bytes available)”

],

“BackendStatistics”: {

“RemoteCalls”: 4,

“BytesUploaded”: 0,

“BytesDownloaded”: 56945031,

“FilesUploaded”: 0,

“FilesDownloaded”: 3,

“FilesDeleted”: 0,

“FoldersCreated”: 0,

“RetryAttempts”: 0,

“UnknownFileSize”: 0,

“UnknownFileCount”: 0,

“KnownFileCount”: 1167,

“KnownFileSize”: 30546185811,

“LastBackupDate”: “2021-12-28T10:14:56-03:00”,

“BackupListCount”: 1,

“TotalQuotaSpace”: 1099511627776,

“FreeQuotaSpace”: 0,

“AssignedQuotaSpace”: -1,

“ReportedQuotaError”: true,

“ReportedQuotaWarning”: false,

“MainOperation”: “Test”,

“ParsedResult”: “Success”,

“Version”: “2.0.6.3 (2.0.6.3_beta_2021-06-17)”,

“EndTime”: “0001-01-01T00:00:00”,

“BeginTime”: “2021-12-29T13:22:21.9948411Z”,

“Duration”: “00:00:00”,

“MessagesActualLength”: 0,

“WarningsActualLength”: 0,

“ErrorsActualLength”: 0,

“Messages”: null,

“Warnings”: null,

“Errors”: null

}

The other notebook that I tested configuring to an external hard drive worked perfectly well, so from my point of view it has to be related to upload the backups to One Drive V2

Hope this can help

Regards,

Sebastian

ts678 · December 30, 2021, 12:56pm

Thanks for the info. The problem for me is that it reports correctly with the only OneDrive I have, plus I didn’t develop the query code (or any substantial part of Duplicati for that matter). I’m not sure what other Duplicati staff may have something to test with. There is no shared pool of destinations as far as I know. It’s personal collections. It’s also possible that it only happens on your OneDrive, as I see no other reports (feel free to do searching on your own, both in the forum and in Issues). I couldn’t find one showing the strange behavior of:

“TotalQuotaSpace”: 1099511627776,
“FreeQuotaSpace”: 0,

Presumably the 1 TB is from OneDrive. I suppose you could test a system and raise it to see if it changes, or if by some chance the FreeQuotaSpace stops being 0. All your reports posted are 0. What value do you get on a backup that is not showing the problem? Does it report that 1 TB along with non-zero free quota?

Looking at the code, I don’t see a way to turn off the quota error on a destination that “should” return quota.

This could possibly be examined by you through a relatively complicated process of seeing what OneDrive returned, directly with network tracing from .NET Framework. It’s not tough to enable, but tough to interpret, and posting a file will publish sensitive authentication data. Still, this path might be easier than a debugger.

Another messy debug option might be to try to have you PM credentials of an empty for-debug account to the expert on this area (not me…) who can then probably use a code debugger or similar to look into this.

A messy workaround might be to use rclone Microsoft OneDrive support to get access, then use Duplicati Rclone storage type to run your debugged rclone configuration (using rclone itself) to accomplish backups.

I suppose I should inquire what storage type you are currently using for OneDrive, and what exactly variety of OneDrive you are on. OneDrive is kind of a marketing name, and underlying technologies may vary a lot. Storage Providers has a long list of various OneDrive/SharePoint that Duplicati has at one time supported.

There might be another that works better, but the fact that only some of the users fail (correct?) is strange, however looking at what sorts of quota statistics the working users are seeing might still help isolate issue.

I did find one surprising result on my statistics-look-perfect OneDrive test. I’m not sure why I’m not getting warned about the 98% full situation. I might look at that, but yours is different – your statistics are wrong at least for some users, and the pattern to that is not clear to me. Maybe you can see if you can identify one.

If you can find another tool that can detect quotas remotely (I did not, in a not-super-heavy search), try it…