Initial backup to USB then WebDAV

Interesting solution!

Please let us know if it works for you next time. :slight_smile:

Hi guys,
short summary of the last weeks using the proclaimed solution.

It works, but it takes forever to compare the client and server side. Surely this is the mentioned problem when using webdav with these many single filesā€¦

Is there something new about the possible option to split the backup chunks into subfolders?

1 Like

At least the ā€œit worksā€ part is good news. :slight_smile:

Not that Iā€™m aware ofā€¦but that doesnā€™t mean somebody isnā€™t working on it.

i have tried the local backup to USB, then copying those files to my nextcloud. Changing the duplicati parameters to webdav instead of local file, tests ok, but then lists all 3600 files missing. Clearly, Iā€™m missing a step. itā€™s almost as if the backup database is storing the local path to the files, and then when they are moved to an external webdav location, they are all ā€˜missingā€™. Thatā€™s a complete WAG on my part, as Iā€™m neither a webdav or database person, but its the only thing I could come up with.

This isnā€™t the caseā€¦ I have migrated between different back ends a few times. As long as you are very careful about transfering the files and making sure the paths are correct, it should be fine.

rclone is the best command line utility (in my opinion) for syncing to/from/between cloud storage providers. It does support WebDAV/Nextcloud: WebDAV

Iā€™d recommend testing the process/workflow with a tiny backup. Once you get the details figured out you can do it with your main backup. (Make sure each backup has its own unique folder of course.)

Doc - Probably wasnā€™t clear in my original query. Iā€™m not changing back-ends, Iā€™m simply doing sneaker-net to avoid a transfer speed of 1.5 mbps!!

I have a remote system with 300G of data. Using duplicati on the remote system with a webdav destination would take 30 days (!!)
So I cable in a SSD locally, and set my duplicati destination to be a NTFS file system on that SSD. A day later, I collect the SSD and mount it on my nextcloud server, copying all of the files into the nextcloud account for that remote user.

Remotely connect into the source system and change the destination to my webdav location, and all 3600 files are ā€˜missingā€™. The only option Iā€™ve found is to delete the remote database and recreate it, which while somewhat faster than uploading all of the dblocks, is still painful.

Iā€™ve read several stories online where what Iā€™m trying to do seems easy. Thus far, this ā€˜easyā€™ exercise has defeated me.

To me it sounds like you are seeding the initial backup.

I have done the same thing a few times using a USB drive and then physically moving the drive to a remote location so I could quickly copy the data to the intended destination. Then I just adjust the backup set on the Duplicati machine to target the remote storage instead of the (formerly) local USB drive.

This is still a back end migration in a way - similar steps are involved.

If after relocating the backup data and reconfiguring the backup job, if it complains about missing remote data it still sounds to me like itā€™s not looking in exactly the right path. But if that were the case I would think a database rebuild would fail.

I must be missing somethingā€¦

well the database rebuild is taking forever and I have exactly zero confidence that it is going to work in any case. Iā€™ve just been schooled on how to use some of the commandline tools (another frustration solved!), so let me play with those and Iā€™ll get back to you.
Thank you!

Doc - I thought I had it figured out - the missing piece was running the test on the command line to verify the backend

destination. That worked on one of my sneaker-net ops, in spite of needing a very specific nextcloud directory name. Regardless, the next

attempt failed like all of the others:

At the customer site, i set up a duplicati job to backup to a local SSD/USB drive. That finished fine, with 2200+ files on it including dlist and dindex files.

After sneaker-netting the files back to my shop and copying into my nextcloud:

I logged on remotely to the customer system, change the destination parameters on the duplicati config file, extracted this command and ran it:

ā€œC:\Program Files\Duplicati 2\Duplicati.CommandLine.exeā€ test ā€œwebdavs://{nextcloud URL}/remote.php/dav/files/{cust.name}?auth-username={username}&auth-password={pw}ā€ --dbpath=ā€œC:\Users{cust.name}\AppData\Local\Duplicati\VIPBKRETPQ.sqliteā€ --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase="{secret}" --disable-module=console-password-input

yields

May 7, 2020 2:32 AM: Missing file: duplicati-20200506T052415Z.dlist.zip.aes

~2200 more like this

Missing file: duplicati-ibd7f315925e845c4a4088d4a7f14c4ec.dindex.zip.aes

Missing file: duplicati-i12f1fca75f1f4405abc0c56cf3cda17d.dindex.zip.aes

Missing file: duplicati-bd721235088c744c0aecd484243d7a438.dblock.zip.aes

Missing file: duplicati-ifcc3bdd1533b45e7a82fa731f8405b4c.dindex.zip.aes

Missing file: duplicati-b2a9548eb7ba54465a17c50ca57a80ce4.dblock.zip.aes

I do an ā€˜ls -lā€™ on these and select others in my nextcloud location where I loaded the sneaker-net files:

-rwxrwxrwx 1 root root 14406717 May 5 22:26 duplicati-20200506T052415Z.dlist.zip.aes

-rwxrwxrwx 1 root root 26589 May 5 01:03 duplicati-ibd7f315925e845c4a4088d4a7f14c4ec.dindex.zip.aes

-rwxrwxrwx 1 root root 27021 May 5 01:03 duplicati-i12f1fca75f1f4405abc0c56cf3cda17d.dindex.zip.aes

-rwxrwxrwx 1 root root 52338877 May 5 01:03 duplicati-bd721235088c744c0aecd484243d7a438.dblock.zip.aes

-rwxrwxrwx 1 root root 27277 May 5 01:03 duplicati-ifcc3bdd1533b45e7a82fa731f8405b4c.dindex.zip.aes

-rwxrwxrwx 1 root root 52338413 May 5 01:03 duplicati-b2a9548eb7ba54465a17c50ca57a80ce4.dblock.zip.aes

Why canā€™t the duplicati process see these files? What other tests or commandline arguments can I use to help troubleshoot this.

I did not run any commandline stuff on the remote system against the windows local filesystem to see if the problem was there and I just

transferred the headache. Next time I will be sure to do that.

Frustration continues because whether it works or not (at least at this point) appears random.

Thanks for your help

Rick

Iā€™d probably try using a 3rd party WebDAV client, just to verify that the expected files are visible at the URL youā€™re using.

In addition, you can see what Duplicati sees (which might be nothing ā€“ then question becomes why) at <job> --> Show log --> Remote to find the list at start of backup. Click on it, to see what it sees there.