I have roughly 5TB of slow changing data i wish to back up. I really like the idea of deduplication and i’m a big fan of FOSS. That’s how i got to duplicati. Just before, i played around with duplicity which is also very nice but doesn’t do the deduplication.
Alas, here is my storage plan, for which i’m wondering if i can get this up and running with duplicati:
When installing duplicati on my main machine, i want to write a full backup to a temporarily connected 6TB hard-disk. I want it to be a full backup, so restorable and all that, not just a subset of backup files. Next, i want the following/incremental backups to go to B2/SSH/Onedrive/… somewhere online.
- Because now i can keep the file (like old tape archives) somewhere in a fault at some airgapped space, safe from intrusion.
- I can load the disk without massive wait times (it took me months of continuous upload to store it all on backblaze backup and i don’t feel like doing that again)
- A single hard drive is a lot cheaper than a 2 year online 5TB storage plan.
- I can choose to move all the files from online to the database to “merge” the two and have less data online to cut cost. - i would consider this really premium
I can really easily make this work with duplicity, but to avoid loads of duplicate data when i move a directory or maintain those moves by hand in the backup, i would really appreciate dupliciti’s abilities.
I’ve tried to emulate it with a docker-install and some source files to backup. But as soon as i remove some dblock files from the destination (to emulate those on a disk and not present online), duplicati is smart enough to recognize that those files are missing, and it won’t continue the backup. Is there anyway i can force the continuation of the backup? I’ve seen the
allow-missing-source but i need a
allow-missing-dblock file thing.
(I have tried to find a similar question online but could not find one, if there is one please excuse me, a pointer would be appreciated)
After tinkering some more, i discovered the
no-backend-verification option, which does have the desired effect, but it might also have some undesired side-effects for what i want. My bet is that this would not work when uploading files online, because the index files would need to be read on each backup?
Thanks for writing wonderful software, and any help!