Access Denied when backing up to wasabi

Will2k_nz · February 1, 2023, 9:15am

Hey all

I am having a issue with one of my backups to wasabi. There are 2 backups which are backing up to one of my wasabi bucket. One is working fine but the 2nd one keeps come with “Failed: Access Denied” (Please see full log at bottom of this post)

I was trying to get Bucket level access to work but didn’t have much luck. Which I believe might be casing this issue.

Tonight I created a test user at the wasabi end and try this user for the backup with the same out come.

Failed: Access Denied
Details: Amazon.S3.AmazonS3Exception: Access Denied —> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. —> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x00000] in :0
— End of inner exception stack trace —
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003a] in :0
at Amazon.Runtime.Internal.HttpHandler1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 --- End of inner exception stack trace --- at Duplicati.Library.Main.BackendManager.Delete (System.String remotename, System.Int64 size, System.Boolean synchronous) [0x0005c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable1[T] protectedFiles) [0x0061a] in :0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable1[T] protectedFiles) [0x00000] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0 at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action1[T] method) [0x0011c] in :0

Log data:
2023-02-01 21:39:21 +13 - [Warning-Duplicati.Library.Main.BackendManager-DeleteFileFailure]: Failed to recover from error deleting file duplicati-b286efeab17dc4be7a1315f181c395a1e.dblock.zip.aes
System.NullReferenceException: Object reference not set to an instance of an object
at Duplicati.Library.Main.BackendManager.ThreadRun () [0x003e3] in :0
2023-02-01 21:39:21 +13 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Amazon.S3.AmazonS3Exception: Access Denied —> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. —> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x00000] in :0
— End of inner exception stack trace —
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003a] in :0
at Amazon.Runtime.Internal.HttpHandler1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 --- End of inner exception stack trace --- at Duplicati.Library.Main.BackendManager.Delete (System.String remotename, System.Int64 size, System.Boolean synchronous) [0x0005c] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable1[T] protectedFiles) [0x0061a] in :0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in :0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in :0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00452] in :0

gpatel-fr · February 1, 2023, 11:20am

Hello

you have a very clear error message here, could you try to follow up on it by using some S3 explorer tool like maybe mc (on Linux) to connect from the computer having Duplicati installed, with the same credentials, to explore the backend and find this file, and if it can be found, try to delete (possibly saving it first somewhere in any case) or rename it and see if you get a more explicit error message ?

Will2k_nz · February 2, 2023, 9:04am

thanks for your reply

regrading the file that is mention in the error message i was able to located it on the wasabi end using Wasabi Explorer for Cloud Storage. I then delete the file and re try the backup it stated to count the files then failed again. It now mentions a different file Failed to recover from error deleting file duplicati-b536d07c4285b49e19e90ef40cd537b86.dblock.zip.aes

Failed: Access Denied
Details: Amazon.S3.AmazonS3Exception: Access Denied —> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. —> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x00000] in :0
— End of inner exception stack trace —
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003a] in :0
at Amazon.Runtime.Internal.HttpHandler1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 --- End of inner exception stack trace --- at Duplicati.Library.Main.BackendManager.WaitForEmpty (Duplicati.Library.Main.Database.LocalDatabase db, System.Data.IDbTransaction transation) [0x00067] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.CompactHandler.DoDelete (Duplicati.Library.Main.Database.LocalDeleteDatabase db, Duplicati.Library.Main.BackendManager backend, System.Collections.Generic.IEnumerable1[T] deleteableVolumes, System.Data.IDbTransaction& transaction) [0x00040] in :0
at Duplicati.Library.Main.Operation.CompactHandler.DoCompact (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Boolean hasVerifiedBackend, System.Data.IDbTransaction& transaction, Duplicati.Library.Main.BackendManager sharedBackend) [0x004fb] in :0
at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBackend, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x003a2] in :0
at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in :0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01048] in :0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in :0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in :0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in :0

Log data:
2023-02-02 21:53:29 +13 - [Warning-Duplicati.Library.Main.BackendManager-DeleteFileFailure]: Failed to recover from error deleting file duplicati-b536d07c4285b49e19e90ef40cd537b86.dblock.zip.aes
System.NullReferenceException: Object reference not set to an instance of an object
at Duplicati.Library.Main.BackendManager.ThreadRun () [0x003e3] in :0
2023-02-02 21:53:29 +13 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Amazon.S3.AmazonS3Exception: Access Denied —> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) Forbidden. —> System.Net.WebException: The remote server returned an error: (403) Forbidden.
at System.Net.HttpWebRequest.GetResponseFromData (System.Net.WebResponseStream stream, System.Threading.CancellationToken cancellationToken) [0x00146] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.RunWithTimeoutWorker[T] (System.Threading.Tasks.Task1[TResult] workerTask, System.Int32 timeout, System.Action abort, System.Func1[TResult] aborted, System.Threading.CancellationTokenSource cts) [0x000f8] in <33b19a7ad5234d94abf4fd9b47566616>:0
at System.Net.HttpWebRequest.GetResponse () [0x00016] in <33b19a7ad5234d94abf4fd9b47566616>:0
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x00000] in :0
— End of inner exception stack trace —
at Amazon.Runtime.Internal.HttpRequest.GetResponse () [0x0003a] in :0
at Amazon.Runtime.Internal.HttpHandler1[TRequestContent].InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00073] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.RedirectHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.Unmarshaller.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <79796cc3f14a4a2d81133207810e557f>:0 at Amazon.Runtime.Internal.PipelineHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x0000e] in <e28e89f25c1649a69062a2c53f89d718>:0 at Amazon.Runtime.Internal.ErrorHandler.InvokeSync (Amazon.Runtime.IExecutionContext executionContext) [0x00000] in <e28e89f25c1649a69062a2c53f89d718>:0 --- End of inner exception stack trace --- at Duplicati.Library.Main.BackendManager.WaitForEmpty (Duplicati.Library.Main.Database.LocalDatabase db, System.Data.IDbTransaction transation) [0x00067] in <e60bc008dd1b454d861cfacbdd3760b9>:0 at Duplicati.Library.Main.Operation.CompactHandler.DoDelete (Duplicati.Library.Main.Database.LocalDeleteDatabase db, Duplicati.Library.Main.BackendManager backend, System.Collections.Generic.IEnumerable1[T] deleteableVolumes, System.Data.IDbTransaction& transaction) [0x00040] in :0
at Duplicati.Library.Main.Operation.CompactHandler.DoCompact (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Boolean hasVerifiedBackend, System.Data.IDbTransaction& transaction, Duplicati.Library.Main.BackendManager sharedBackend) [0x004fb] in :0
at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBackend, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x003a2] in :0
at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in :0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00da9] in :0

gpatel-fr · February 2, 2023, 9:14am

If you ignore my advice, I think it shows that you are knowing more than me, in this case it’s pointless to try to help you. Good luck.

ts678 · February 2, 2023, 4:20pm

You could possibly ask Wasabi after some further testing. They’re more expert in setting their end up.

Was that the installed program from their Downloads page which sounds like CloudBerry Explorer?

That “seems” like it should be adequate, but did you use the same credentials as Duplicati is using?

We want to make sure that you’re accessing remotely (not on some Wasabi web page) as Duplicati.

You could also test from a Command Prompt using Duplicati tools. First, Export As Command-line to obtain the URL for your folder. Duplicati.CommandLine.BackendTool.exe can let you test operations, however don’t use any names that would be confused with your backup files, typically duplicati-*.

If you prefer, modify the URL to an empty folder and run Duplicati.CommandLine.BackendTester.exe.

Before you start using a particular backend to use as a backup target, you can use the Backend Tester to get an indication of the integrity of that backend.

Will2k_nz · February 8, 2023, 4:38am

Yes it was CloudBerry Explorer.

I used the CloudBerry Explorer to located those files on the Wasabi end.

On the Wasabi end i have change some Policies and have added WasabiFullAccess to the user for that Bucket. It Seems that has fix the issue. Originally i only had read/write set for that user. I find it a bit strange its needs Full Access to allow the backup to work. In my understanding it should only need read/write access, or am i missing something here?

Thanks

Will2k

gpatel-fr · February 8, 2023, 9:07am

possibly the read/write access don’t give the permission to delete objects ?

ts678 · February 8, 2023, 1:37pm

Compacting files at the backend is visible on some of the stacks. Retention options also delete.
Trying to never do a delete risks space usage growing forever. If you want that, it can be tried…