Backing up duplicati backups with rsync to remote drive

Features
1

Hi All,

I wanted to know if there are any caveats with rsyncing the duplicati backup directories (the ones with all the aes volumes) to a remote location to have a 2-tier system with 2 sites. Obviously one risk is that if the duplicati backup directory gets wiped out and the rsync cronjob runs it will then wipe out the remote folder as well. Unfortunately the remote system in this case is behind a firewall that is hard to reverse-tunnel through so it is ideal if the remote system can be the one initating the copy (otherwise I would just have duplicati also backup directly to that system … is that possible btw/).

Any ideas on how people have in general dealt with having a second offsite backup would be welcome!

Best,

Gabe

1 Like
2

Hi Gabe,

I have the same thoughts and concerns.

On one hand rsync seems to be very robust (line disconnections, IP changes by provider on both locations, … do not disturb rsync).
On the other hand rsync might delete files :frowning:

How robust is duplicati via transfering data to a remote location?

Especially when working with “asynchronous-upload-limit” (and “asynchronous-upload-folder”), is there a resume function?

(When using “asynchronous-upload-limit”, would the next duplicati start, while the async buffer gets transferred?)

(What about “Having duplicati store on a rsync server”? Is this an idea to think about?)

(Should I open a new thread?)

Best regards,
Ingo

3

Personally, the “all files in source were deleted so let’s sync that to the destination” problem is the biggest one I’m aware of. A few people have shown concern that a sync could happen while Duplicati is still uploading a file, but I suspect either rsync will be smart enough to not sync a file that’s in use or (worst case) it will sync the half-written file then sync the rest of it on the next, um, sync.

For some people that might be a problem (having a window of failed restore possibility) but for me it’s small window with a small potential number of affected file versions.

A do believe a few users have set up rsync or Syncthing to do what you’re describing so it’s certainly not unheard of. The biggest thing I think people should be aware of is that IF you have to restore from the “2nd level” backup location then you’ll either have to make it exposed so that Duplicati can see it as a destination OR you’ll have to manually move/copy all the files to a somewhere the Duplicati can see.

I haven’t used rsync personally, but I suspect there might be an option to tell it to NOT sync file deletes, which should help with that worry. However that could also cause issues if you need to restore from the “2nd level” location as Duplicati will likely see old files it thought it had cleaned up and complain about them.

I think I might be mis-understanding your question about “a resume function”… regarding the asynchronous-upload parameters

–asynchronous-upload-folder= (default)
The pre-generated volumes will be placed into the temporary folder by default, this option can set a different folder for placing the temporary volumes, despite the name, this also works for synchronous runs

–synchronous-upload-limit=4 (default)
When performing asynchronous uploads, Duplicati will create volumes that can be uploaded. To prevent Duplicati from generating too many volumes, this option limits the number of pending uploads. Set to zero to disable the limit

Duplicati jobs run one-at-a-time, so if the current job isn’t done uploading everything then the next job won’t start.

I thought I saw somewhere that somebody was trying to add rsync as a destination type, but I can’t seem to find the post (here or on GitHub) at the moment. :frowning:

4

Resume the upload after a connection breakdown, which stopped the upload.

Yes, breakdowns are quite seldom nowadays, but just this week they worked on the electricity lines in our street for three hours. (I believe, duplicati does not work with timeouts of hours :wink:

Thank you!

5

Thanks for the clarification. I think the best you can do us what’s mentioned in the post below, but be aware that if your connection is REALLY bad (or down for a long while) it could make a normal 30 min. backup appear to take 30 hours (due to the retries).

And of course, if the computer is shut down (not sleep, but actual power off) the backup will be aborted - but will pick up where it left off when the next scheduled job runs kicks off.

6

Hi Gabe,

I’m currently running an off-site backup via a VPN to a remote second PC. The VPN is LogMeIn Hamachi, which is free up to 5 PCs in a network. Although I think the “run Hamachi as a service” mode is only available in the paid plans.
I just share a folder, with the right read and write permissions, on an external HDD attached to the second remote PC. Then I configure the backup to use that folder as a destination, using the Hamachi-assigned IP of the second PC (e.g. \\25.xxx.xxx.xxx\Shared Folder)
I use an external HDD so as to be able to run the initial backup on-site, configuring Duplicati to use a local destination for the first backup, then take the drive from the site of the first PC to the remote site of the second PC and run the second and subsequent backups through the VPN, configuring Duplicati on the first PC to use the remote destination

Hope it helps. Let me know if you have any questions about my backup configuration

7
  • Resources & speed. Rsync synchronizes files by checking bit by bit. The comparison bit by bit is not required as duplicati does never change existing files. This means, you are using rsync actually just to compare the list of files and to transfer files. The main feature to check and update the content of files is not used. If you cannot turn this off, this will slow down the whole process and eat resources. What is the rsync transfer speed and how fast can duplicati be via ftp, SSH, …?
  • Robustness. Duplicati updates its database when all files have been uploaded properly. This makes it quite robust. Even if the process is interrupted, it does not break the integrity of files. If the source server gets “killed” while rsync is working, this might leave the target data in a weird state so that the content of a file might be broken. E.g. you could have synchronized the meta data for the backup, but some content files are missing. It could also mean that content files have be updated partially.