Interesting solution!
Please let us know if it works for you next time.
Interesting solution!
Please let us know if it works for you next time.
Hi guys,
short summary of the last weeks using the proclaimed solution.
It works, but it takes forever to compare the client and server side. Surely this is the mentioned problem when using webdav with these many single filesā¦
Is there something new about the possible option to split the backup chunks into subfolders?
At least the āit worksā part is good news.
Not that Iām aware ofā¦but that doesnāt mean somebody isnāt working on it.
i have tried the local backup to USB, then copying those files to my nextcloud. Changing the duplicati parameters to webdav instead of local file, tests ok, but then lists all 3600 files missing. Clearly, Iām missing a step. itās almost as if the backup database is storing the local path to the files, and then when they are moved to an external webdav location, they are all āmissingā. Thatās a complete WAG on my part, as Iām neither a webdav or database person, but its the only thing I could come up with.
This isnāt the caseā¦ I have migrated between different back ends a few times. As long as you are very careful about transfering the files and making sure the paths are correct, it should be fine.
rclone is the best command line utility (in my opinion) for syncing to/from/between cloud storage providers. It does support WebDAV/Nextcloud: WebDAV
Iād recommend testing the process/workflow with a tiny backup. Once you get the details figured out you can do it with your main backup. (Make sure each backup has its own unique folder of course.)
Doc - Probably wasnāt clear in my original query. Iām not changing back-ends, Iām simply doing sneaker-net to avoid a transfer speed of 1.5 mbps!!
I have a remote system with 300G of data. Using duplicati on the remote system with a webdav destination would take 30 days (!!)
So I cable in a SSD locally, and set my duplicati destination to be a NTFS file system on that SSD. A day later, I collect the SSD and mount it on my nextcloud server, copying all of the files into the nextcloud account for that remote user.
Remotely connect into the source system and change the destination to my webdav location, and all 3600 files are āmissingā. The only option Iāve found is to delete the remote database and recreate it, which while somewhat faster than uploading all of the dblocks, is still painful.
Iāve read several stories online where what Iām trying to do seems easy. Thus far, this āeasyā exercise has defeated me.
To me it sounds like you are seeding the initial backup.
I have done the same thing a few times using a USB drive and then physically moving the drive to a remote location so I could quickly copy the data to the intended destination. Then I just adjust the backup set on the Duplicati machine to target the remote storage instead of the (formerly) local USB drive.
This is still a back end migration in a way - similar steps are involved.
If after relocating the backup data and reconfiguring the backup job, if it complains about missing remote data it still sounds to me like itās not looking in exactly the right path. But if that were the case I would think a database rebuild would fail.
I must be missing somethingā¦
well the database rebuild is taking forever and I have exactly zero confidence that it is going to work in any case. Iāve just been schooled on how to use some of the commandline tools (another frustration solved!), so let me play with those and Iāll get back to you.
Thank you!
Doc - I thought I had it figured out - the missing piece was running the test on the command line to verify the backend
destination. That worked on one of my sneaker-net ops, in spite of needing a very specific nextcloud directory name. Regardless, the next
attempt failed like all of the others:
At the customer site, i set up a duplicati job to backup to a local SSD/USB drive. That finished fine, with 2200+ files on it including dlist and dindex files.
After sneaker-netting the files back to my shop and copying into my nextcloud:
I logged on remotely to the customer system, change the destination parameters on the duplicati config file, extracted this command and ran it:
āC:\Program Files\Duplicati 2\Duplicati.CommandLine.exeā test āwebdavs://{nextcloud URL}/remote.php/dav/files/{cust.name}?auth-username={username}&auth-password={pw}ā --dbpath=āC:\Users{cust.name}\AppData\Local\Duplicati\VIPBKRETPQ.sqliteā --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase="{secret}" --disable-module=console-password-input
yields
May 7, 2020 2:32 AM: Missing file: duplicati-20200506T052415Z.dlist.zip.aes
~2200 more like this
Missing file: duplicati-ibd7f315925e845c4a4088d4a7f14c4ec.dindex.zip.aes
Missing file: duplicati-i12f1fca75f1f4405abc0c56cf3cda17d.dindex.zip.aes
Missing file: duplicati-bd721235088c744c0aecd484243d7a438.dblock.zip.aes
Missing file: duplicati-ifcc3bdd1533b45e7a82fa731f8405b4c.dindex.zip.aes
Missing file: duplicati-b2a9548eb7ba54465a17c50ca57a80ce4.dblock.zip.aes
I do an āls -lā on these and select others in my nextcloud location where I loaded the sneaker-net files:
-rwxrwxrwx 1 root root 14406717 May 5 22:26 duplicati-20200506T052415Z.dlist.zip.aes
-rwxrwxrwx 1 root root 26589 May 5 01:03 duplicati-ibd7f315925e845c4a4088d4a7f14c4ec.dindex.zip.aes
-rwxrwxrwx 1 root root 27021 May 5 01:03 duplicati-i12f1fca75f1f4405abc0c56cf3cda17d.dindex.zip.aes
-rwxrwxrwx 1 root root 52338877 May 5 01:03 duplicati-bd721235088c744c0aecd484243d7a438.dblock.zip.aes
-rwxrwxrwx 1 root root 27277 May 5 01:03 duplicati-ifcc3bdd1533b45e7a82fa731f8405b4c.dindex.zip.aes
-rwxrwxrwx 1 root root 52338413 May 5 01:03 duplicati-b2a9548eb7ba54465a17c50ca57a80ce4.dblock.zip.aes
Why canāt the duplicati process see these files? What other tests or commandline arguments can I use to help troubleshoot this.
I did not run any commandline stuff on the remote system against the windows local filesystem to see if the problem was there and I just
transferred the headache. Next time I will be sure to do that.
Frustration continues because whether it works or not (at least at this point) appears random.
Thanks for your help
Rick
Iād probably try using a 3rd party WebDAV client, just to verify that the expected files are visible at the URL youāre using.
In addition, you can see what Duplicati sees (which might be nothing ā then question becomes why) at <job> --> Show log --> Remote to find the list
at start of backup. Click on it, to see what it sees there.