Nextcloud Webdav Error - missing files from the remote storage

[greavette]

Hello Forum,

I’m using Duplicati version - 2.0.4.5_beta_2018-11-28
Duplicati is installed on Windows 7 x64 virtual machine.

I’m really trying to get Duplicati to work for me and I’ve been sort of successful but I’m hoping this community can assist me in finding a better solution to my backup through Webdav to Nextcloud. Here’s the backstory on task to test Duplicati with Nextcloud:

Symmcom is a company I go to regularly to help our business with various services. Recently Symmcom has offered a free version of Nextcloud that includes 50 GB of space…FREE! I’ve spoken to their support team who put me in touch with the owner and he confirmed that the 50 GB free space is fully encrypted and backed up. I immediately thought that Duplicati would be a great solution for me to move files (deduped down) to this free storage. Here’s a link to their webpage for the free storage option:

https://cloud.symmcom.com/cart.php?gid=14

For the rest of this forum post, I will reference the following page from the Duplicati documentation on how to create a Duplicati backup job using the GUI: Using the Graphical User Interface - Duplicati 2 User's Manual

Webdav:

My first crack at this was to use Webdav from within Duplicati. I followed the instructions to create my backup job using Webdav and tested my connection to my Nextcloud server and confirmed Duplicati connected successfully. I then ran a test and the backup ran for a long period of time but then failed with an error saying files were missing from the destination server. I logged into my Nextcloud and could see files in the folder I had setup. I then tried to repair (because Duplicati gave me the option to repair) but the error would reoccur. The backup would not complete successfully and the status for this backup job stayed at ‘never’ and would not confirm that the backup completed (even though my Nextcloud server folder had Duplicati files in there). I could not get Webdav to work with Duplicati so I tried another method.

Webdav through windows explorer:

I gave up on Webdav from Duplicati and instead added a network drive to my windows computer using the webdav credentials and assigned a drive letter to my webdav connection to the Symmcom Nextcloud server.
I again followed the instructions to setup a backup job through Duplicati but this time I used the destination drive as the drive letter I assigned to the Nextcloud webdav connection I created in my Windows Explorer. This time when I ran a backup job I again found an error message that files were missing on my Nextcloud server but this time the status for my Duplicati backup did not stay at never, it changed to the date the backup was run. I was making progress finally…but when I added a new test file and ran the job, I again had an error after the backup job finished and again saw a new time/datestamp for the job being finished. So connecting to my Nextcloud server through Windows Explorer webdav worked but kept throwing an error which is a false positive since the backup status seemed to suggest it worked. I again confirmed that my backup files created by Duplicati were in my Nextcloud folder.

Nextcloud Desktop Client

I then tried to backup through the Nextcloud desktop client. I installed the latest version of the Nextcloud Desktop client on the same Windows 7 computer that runs Duplicati and I confirmed that through the Nextcloud Desktop Client I could see all my Symmcom Nextcloud folders. I setup my 3rd test through Duplicati to backup files from my local computer with a destination of the Nextcloud Desktop client and the test connection worked successfully. I ran the first test backup and the Duplicati backup job ran successfully without any error! So I then added a new test file to my source folder and again ran the Duplicati backup job but this time I received a new error that Duplicati couldn’t finish the backup job and to look in the log for errors. But the date of the Last Successful backup did change to the new backup I just ran that said there were errors. I’m including the errors from this last backup here in this post for review by the Duplicati Devs. Hopefully you can find an answer to my issues and see why I’m getting messages on this second backup to Nextcloud through the Nextcloud Desktop client. From what I can see in this log below it looks successful to me.

I’m not against using the NextCloud desktop client, but it does represent a loss of space on my Windows server running Duplicati because i’m using more space on my drive than I’d like. I would really rather use straight Webdav through Duplicati but for some reason Duplicati can’t verify the files properly. Is this a known error?

Any help or information you can provide me would be greatly appreciated. Below is the attached log file from my last test using Nextcloud Desktop folder.

Jan 15, 2019 11:25 PM: Result
DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 0
ExaminedFiles: 5
OpenedFiles: 1
AddedFiles: 1
SizeOfModifiedFiles: 0
SizeOfAddedFiles: 15672518144
SizeOfExaminedFiles: 47274099207
SizeOfOpenedFiles: 15672518144
NotProcessedFiles: 0
AddedFolders: 0
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults: null
DeleteResults:
DeletedSets:
Dryrun: False
MainOperation: Delete
CompactResults: null
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 1/15/2019 11:25:02 PM (1547612702)
BeginTime: 1/15/2019 11:25:01 PM (1547612701)
Duration: 00:00:01.0469688
Messages: [
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (18 bytes),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
…
]
Warnings:
Errors:
BackendStatistics:
RemoteCalls: 8
BytesUploaded: 46173191
BytesDownloaded: 525505255
FilesUploaded: 3
FilesDownloaded: 3
FilesDeleted: 0
FoldersCreated: 0
RetryAttempts: 0
UnknownFileSize: 0
UnknownFileCount: 0
KnownFileCount: 21
KnownFileSize: 3331860049
LastBackupDate: 1/15/2019 11:15:31 PM (1547612131)
BackupListCount: 3
TotalQuotaSpace: 161058123776
FreeQuotaSpace: 94836137984
AssignedQuotaSpace: -1
ReportedQuotaError: False
ReportedQuotaWarning: False
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
Messages: [
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (18 bytes),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
…
]
Warnings:
Errors:
RepairResults: null
TestResults:
MainOperation: Test
Verifications: [
Key: duplicati-20190116T041531Z.dlist.zip.aes
Value: ,
Key: duplicati-i0b8f2211e4a547deb3bc12050a595b45.dindex.zip.aes
Value: ,
Key: duplicati-babac74fcca3a482d890cdbbcc8625cf9.dblock.zip.aes
Value:
]
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 1/15/2019 11:25:12 PM (1547612712)
BeginTime: 1/15/2019 11:25:05 PM (1547612705)
Duration: 00:00:07.3355754
Messages: [
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (18 bytes),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
…
]
Warnings:
Errors:
ParsedResult: Success
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 1/15/2019 11:25:12 PM (1547612712)
BeginTime: 1/15/2019 11:15:31 PM (1547612131)
Duration: 00:09:41.7414868
Messages: [
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: (),
2019-01-15 23:15:31 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (18 bytes),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
2019-01-15 23:25:00 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ba5ef8b48f02746fb8c9ccec220ebb193.dblock.zip.aes (39.87 MB),
…
]
Warnings:
Errors:

[greavette]

A bit more detail for the devs to review on my test of using Webdav from within Duplicati. I tried to send 3 small files to my Nextcloud server at Symmcom and it failed with a files missing message. Here are the log data from the server:

Jan 17, 2019 7:10 PM: Failed while executing “Repair” with id: 18
Duplicati.Library.Interface.UserInformationException: The backup storage destination is missing data files. You can either enable --rebuild-missing-dblock-files or run the purge command to remove these files. The following files are missing: duplicati-bc61405aa09514ca7a6ab8ee6f06213c3.dblock.zip.aes
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairRemote()
at Duplicati.Library.Main.Operation.RepairHandler.Run(IFilter filter)
at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action1 method) at Duplicati.Library.Main.Controller.Repair(IFilter filter) at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue) Jan 17, 2019 7:09 PM: Failed while executing "Backup" with id: 18 Duplicati.Library.Interface.UserInformationException: Found 3 files that are missing from the remote storage, please run repair at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile) at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification() at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result) at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action1 method)
at Duplicati.Library.Main.Controller.Backup(String inputsources, IFilter filter)
at Duplicati.Server.Runner.Run(IRunnerData data, Boolean fromQueue)

I’m hoping someone can assist me in getting Webdav to work to this Nextcloud server.

Thank you.

[herbert]

Did you find the error, ran into the same issue yesterday and wonder if I could fix this somehow.

Duplicati 2.0.5.1
Nextcloud 19.0.3
Protocol WebDav

[herbert]

I found a hint at

"found <n> files that are missing from the remote storage" investigated a reason for unsuccessful backup

Thank you @Waenna ("found <n> files that are missing from the remote storage" investigated a reason for unsuccessful backup) My slash on the path was giving me this problem. I have uploaded around 100 gb on icedrive for almost 10 hours but its keep saying 733 files missing, error, etc. After reading your post i checked my configuration, there was slash at the end which i removed and saved it. Running the backup again it took 5 minutes and everything ok, amazing. [image]

Figured out that I had two slashes in the path just next to each other folder//folder which worked fine to push the backup to the target but seems that the verification afterwards throws an error.
Removing one of the two slashes fixed the problem for me