Found X files that are missing - All Backups

Support
1

First of all, thank you to all the Devs and Those supporting software like these.

I know this is a popular topic on the forums but I was unable to find any applicable solution after reading all that I could find.

Backup Location
I have an off-site windows server 2016 with 36TB of drives on it I am running Minio service.

Backup Source 1
At my home, I have another windows server 2016 with Duplicati on it that backs up my NAS via UNC paths and sends the backup to the Minio S3 box.

I set this up about a year ago and after an initial local seed, I was able to remotely backup about 6TB worth of data for about 10 months with almost zero issues.

One day I started getting some errors, possibly from another unrelated issue, and eventually, my backups became out of sync and corrupt after many attempts to repair and delete the database I eventually decided to scrap all of my backup files and start clean.

This time I am not doing a local seed and no matter how many different ways I try to do my backups I always get the dreaded “Found X files that are missing from the remote storage, please run repair”

On my remote host, I have updated Minio, recreated the service from scratch, new configs, new keys and wiped out all of the legacy files.

On my local machine, I tried creating a new test backup config with only a few hundred MB of files and it will run, I can see the remote files appearing and it will end with a message “Found 2 files that are missing from the remote storage, please run repair”. I can verify that indeed the files are missing.

It will say there should be 17 files and there will be 15.

On my remote host I had windows DeDupe enabled, which had never caused issues before, but just to be sure I disabled deduplication on the volume.

Backup Source 2
I tried installing Duplicati on another local desktop just to be sure it wasn’t an issue with my Duplicati install on my server. I again created a new test backup with a few hundred MB of files and again it will create the bucket and transfer all but a few files and then fail with the error “Found X files that are missing from the remote storage, please run repair”

Clicking repair does nothing visible.

I tried to enable rebuild-missing-dblock-files on my test backup and it still fails.

Version Information

Minio Version:
2020-05-06T23:23:25Z

Local Server Duplicati Version:
2.0.5.104_canary_2020-03-25

Local Desktop Duplicati Version:
2.0.5.1_beta_2020-01-18

2

Welcome to the forum @MrKuenning

Found 3 files that are missing from the remote storage, please run repair has troubleshooting steps, however your case seems only partly failing. Still need a log to see what Duplicati thinks it uploaded. Maybe it’s not having easy uploads. You can also check for “RetryAttempts” in job’s “Complete log”.

If Duplicati thinks it uploaded it successfully, and it didn’t show up, my first guess would be to try this

image

if you haven’t already. I don’t run MinIO or S3, but possibly MinIO’s client works better with its server.

3

So this may be solved

Ironically your suggestion was what pointed me in the right direction but for the wrong reason.

A year ago when my backups were working fine, the option to use the Minio SDK was not an option and yet it worked great. In my recent testing, I have tried the Minio setting to no avail.

However I just tried swapping my test backup on both local systems back to the amazon S3 setting and they both finished successfully.

I am fairly sure that it wasn’t the lone cause in my backup woes, as I only recently swapped to that setting, but more likely I fixed my original issue and introduced a new one with that setting.

Most of the posts I have found with the error are situations where duplicati is getting a list showing the files as missing but the files aren’t actually missing. In my case the files are actually missing.

Makes me wonder what that Minio setting does different, and why it seems to explicitly fail with a Minio endpoint.

I am testing with one my larger backup and will update on the success or fail.

4

Previously mentioned troubleshooting steps (e.g. getting a log at Retry level) might give some clue.
The MinIO client is a recent addition to fix a particular problem (but not one involving MinIO server).

5

So the backups are still not succeeding.
My small few hundred MB test backups are successful. But my hundreds of GB backups fail over and over again.

At first, they would run for a few hours and then fail with this error

Failed: One or more errors occurred.
Details: System.AggregateException: One or more errors occurred. —> System.AggregateException: An existing connection was forcibly closed by the remote host —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

Then I added “retry delay 5 seconds” and “number of retries 1000”

This would allow the backup to run longer but eventually it now fails with this error.

Failed: One or more errors occurred.
Details: System.AggregateException: One or more errors occurred. —> System.AggregateException: Unable to read data from the transport connection: The connection was closed. —> System.IO.IOException: Unable to read data from the transport connection: The connection was closed.

I keep hitting retry, every few hours in hopes that once the initial data is finally there, the deltas will succeed. But I am unable to get it to finish.

6

Duplicati.CommandLine.BackendTester.exe can be pointed to an empty folder to test reliability.
There are a lot of configuration variables. Export As Command-line can give URL you can edit.
Duplicati.CommandLine.BackendTool.exe gives you manual control if you ever prefer that way.

I wasn’t entirely clear on whether the Amazon AWS SDK works successfully. I see you had said:

The MinIO SDK is probably MinIO Client SDK for .NET. AWS SDK for .NET is probably the first one.

Are there any logs on the MinIO server which might give clues why it’s closing connections on you?

7

I am running into roadblocks everywhere. Sigh. This all used to just work so well.

Duplicati.CommandLine.BackendTester.exe can be pointed to an empty folder to test reliability.

I tried to use the backed tester. I ran the command like this:

Duplicati.CommandLine.BackendTester.exe s3://username:password@address/bucketname/newfoldername

I get the error.

Starting run no 0
Unittest failed: Amazon.S3.AmazonS3Exception: The specified bucket is not valid. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (400) Bad Request. ---> System.Net.WebException: The remote server returned an error: (400) Bad Request.
   at System.Net.HttpWebRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpRequest.GetResponse()
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception)
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception)
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext)
   at Amazon.Runtime.AmazonServiceClient.Invoke[TResponse](AmazonWebServiceRequest request, InvokeOptionsBase options)
   at Duplicati.Library.Backend.S3AwsClient.<ListBucket>d__12.MoveNext()
   at Duplicati.Library.Backend.S3.<List>d__29.MoveNext()
   at Duplicati.Library.Interface.BackendExtensions.TestList(IBackend backend)
   at Duplicati.CommandLine.BackendTester.Program.Run(List`1 args, Dictionary`2 options, Boolean first)
   at Duplicati.CommandLine.BackendTester.Program.RealMain(String[] _args)

I wasn’t sure how you reference buckets. All of the documentation referenced FTP, and I couldn’t find any examples online on how to run it against an S3 target.

I wasn’t entirely clear on whether the Amazon AWS SDK works successfully. I see you had said:

The MinIO SDK is probably MinIO Client SDK for .NET. AWS SDK for .NET is probably the first one.

Sorry if my wording was confusing.

On my test backup of a few hundred MB of files:

  • When I select “Minio SDK” the backup will finish and then tell me I am missing two or three files, when I check the remote end I can see that those files are in fact missing.

  • When I set my test backup to “Amazon AWS SDK” The backup will finish successfully.

However, my larger backups with the same settings will run for 15 minutes to a few hours before failing saying the connection was closed.

Are there any logs on the MinIO server which might give clues why it’s closing connections on you?
I spent several hours trying to get any sort of log file or console logging and short of building a new database I am coming up blank. You are supposed to be able to run an “mc admin trace” command to see the activity logging, but it comes up empty for me.

I own the hosts and the firewalls at both locations, and while I would like to get this setup working, I am not married to Minio. If there is a better windows based target I can use with duplicate I am open to ideas.

I am just confused why it worked so well for a year.

8

Use your current backup as an example URL, and edit folder to empty one: