Errors on three Windows Server 2012 R2

ollibraun · February 15, 2019, 11:18am

Hi,

I have trouble on three different servers. I have installed Duplicati 2.0.4.5 new on this machines. The jobs working with WebDAV, running for a few days (17.000 files and 450GB are on the backup location) ending with “Last successful backup: Never - Run now”

In the Logs I have found the following text:

Feb 14, 2019 7:50 PM: Failed while executing "Backup" with id: 1

Duplicati.Library.Interface.UserInformationException: Found 5248 files that are missing from the remote storage, please run repair
bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
bei Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.b__0(BackupResults result)
bei Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
bei Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

What should I do? (I have tried a lot of things in the last weeks, maybe we can do it together?)

ollibraun · February 15, 2019, 11:34am

I have deleted the database and started database -> repair. After a few seconds:

Feb 15, 2019 12:30 PM: Failed while executing "Repair" with id: 1
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
   bei Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bei Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bei Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
   bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.Repair(IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Same every time: Duplicati can save hundreds of GB to the remote location, but when I try “repair” it says: “No files were found at the remote location, perhaps the target url is incorrect?”

ollibraun · February 15, 2019, 11:49am

Ok, I have now selected only ten files for backup.
I have removed the folder on the backup location (WebDAV).
I have closed duplicati and have deleted every file on C:\Users\admin\AppData\Local\Duplicati - only file “Duplicati-server.sqlite” is left.

Run duplicati again. No successfull backup.

But there are three new files on the - new created - backup location:

duplicati-20190215T113801Z.dlist.zip                    1,57 KB ZIP Other File (VLC)                Heute 12:39    -------
duplicati-bbb6f2811acaf47b6a57a4eb522b74c07.dblock.zip  15,4 MB ZIP Other File (VLC)                Heute 12:39    -------
duplicati-i079d6268924844cbb637828a28e95214.dindex.zip  24,1 KB ZIP Other File (VLC)                Heute 12:39    -------

Started “database-> repair”:

15. Febr. 2019 12:44: Failed while executing "Repair" with id: 1
Duplicati.Library.Interface.UserInformationException: No files were found at the remote location, perhaps the target url is incorrect?
   bei Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun(LocalDatabase dbparent, Boolean updating, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bei Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run(String path, IFilter filter, NumberedFilterFilelistDelegate filelistfilter, BlockVolumePostProcessor blockprocessor)
   bei Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal(IFilter filter)
   bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
   bei Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   bei Duplicati.Library.Main.Controller.Repair(IFilter filter)
   bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

Any ideas?

ollibraun · February 16, 2019, 11:18am

Now it’s running with S3 (QNAP-NAS). Same problem on all three servers:

Failed: Der Prozess kann nicht auf die Datei zugreifen, da sie bereits von einem anderen Prozess verwendet wird.

Details: System.IO.IOException: Der Prozess kann nicht auf die Datei zugreifen, da sie bereits von einem anderen Prozess verwendet wird.
bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
bei System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
bei Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.b__0(BackupResults result)
bei Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)

Log data:
2019-02-16 10:31:53 +01 - [Warning-Duplicati.Library.Main.Operation.BackupHandler-BackendVerifyFailedAttemptingCleanup]: Backend verification failed, attempting automatic cleanup
Amazon.S3.AmazonS3Exception: Could not parse the specified URI —> Amazon.Runtime.Internal.HttpErrorResponseException: Der Remoteserver hat einen Fehler zurückgegeben: (400) Ungültige Anforderung. —> System.Net.WebException: Der Remoteserver hat einen Fehler zurückgegeben: (400) Ungültige Anforderung.
bei System.Net.HttpWebRequest.GetResponse()
bei Amazon.Runtime.Internal.HttpRequest.GetResponse()
— Ende der internen Ausnahmestapelüberwachung —
bei Amazon.Runtime.Internal.HttpRequest.GetResponse()
bei Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext)
bei Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext)
bei Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext)
bei Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext)
bei Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext)
— Ende der internen Ausnahmestapelüberwachung —
bei Duplicati.Library.Main.BackendManager.List()
bei Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
2019-02-16 10:32:23 +01 - [Warning-Duplicati.Library.Main.Operation.RepairHandler-FailedToReadLocalDatabase]: Failed to read local db C:\Windows\system32\config\systemprofile\AppData\Local\Duplicati\73686582906775696684.sqlite, error: database is locked database is locked System.Data.SQLite.SQLiteException (0x80004005): database is locked database is locked
bei System.Data.SQLite.SQLite3.Step(SQLiteStatement stmt)
bei System.Data.SQLite.SQLiteDataReader.NextResult()
bei System.Data.SQLite.SQLiteDataReader…ctor(SQLiteCommand cmd, CommandBehavior behave)
bei System.Data.SQLite.SQLiteCommand.ExecuteReader(CommandBehavior behavior)
bei Duplicati.Library.Main.Database.ExtensionMethods.ExecuteScalarInt64(IDbCommand self, Boolean writeLog, String cmd, Int64 defaultvalue, Object values)
bei Duplicati.Library.Main.Database.LocalDatabase…ctor(IDbConnection connection, String operation)
bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
2019-02-16 10:32:25 +01 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.IO.IOException: Der Prozess kann nicht auf die Datei zugreifen, da sie bereits von einem anderen Prozess verwendet wird.
bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
bei System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()

When I’m trying to reair the database, Duplicati (runs as service with service account) says:

16. Febr. 2019 10:27: Failed while executing "Backup" with id: 1

System.IO.IOException: Der Prozess kann nicht auf die Datei zugreifen, da sie bereits von einem anderen Prozess verwendet wird.
bei System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
bei System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
bei Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
bei Duplicati.Library.Main.Operation.BackupHandler.d__19.MoveNext()
— Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde —
bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
bei CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
bei Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.b__0(BackupResults result)
bei Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method)
bei Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
bei Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

When I’m trying to delete the database, Duplicati says every time, the file is in use. I must reboot the system to delete the database. Maybe that’s the problem?

ts678 · February 20, 2019, 12:20am

Seems pretty extreme. If stopping Duplicati (and checking Task Manager to make sure it’s gone) doesn’t free the database, I wonder what else on the system might be in it? Sysinternals Process Explorer could show you. On the other hand, if you haven’t tried a Duplicati stop, you might be seeing Error “database is locked” when trying to run a backup after a force stop #3445 where Duplicati locks itself out of later work.

Does that mean the WebDAV is QNAP WebDAV? WebDAV in general has been kind of tricky, and QNAP seems to have some additional issues. Interestingly, WebDAV has some special troubleshooting logging.

ollibraun · February 21, 2019, 7:12pm

I had with QNAP-WebDAV the same errors as with QNAP-S3. In the meantime I managed a backup on all three servers. I used S3 for 200GB to 500GB over 10MBit-WAN on QNAP.

I made the storage on QNAP empty.
A backup is done (–snapshot-policy=Required), takes 2 to 4 days, at the end the error comes as above.
I have entered the following parameters:

--no-backend-verification=true
--backup-test-samples=0
--concurrency-max-threads=1

4.Run repair, takes half an hour.

Full backup performed, takes less than an hour. Successful.

Since then the backup works.