In vs Duplicati - 2.0.2.21_canary_2018-03-06 on MacOS I have two jobs running every 2 weeks. The 2nd seems to always take hours Deleting unwanted files.
Form the live / remote log I have copied some lines:
Mar 18, 2018 12:02 PM: put duplicati-i673ba7c02ce7495a933cec85e35f30a5.dindex.zip.aes
Mar 18, 2018 12:02 PM: put duplicati-bf8eec2ad47484c16b3995f267f258aab.dblock.zip.aes
Mar 18, 2018 12:02 PM: get duplicati-ba983890be76e4945a0656fe3c6e1a69c.dblock.zip.aes
Mar 18, 2018 12:02 PM: delete duplicati-i50eb541a4f024b8fbb7a8c983994f49c.dindex.zip.aes
Mar 18, 2018 12:02 PM: delete duplicati-ba95bd33e48784caabe8a6bda577e1ade.dblock.zip.aes
Mar 18, 2018 12:02 PM: delete duplicati-i25a5056f631f4b5fb575eefaae5ff524.dindex.zip.aes
Mar 18, 2018 12:02 PM: delete duplicati-ba8a43c0b8ca7428ba8533ee4df517fa8.dblock.zip.aes
Mar 18, 2018 12:02 PM: put duplicati-i3b0e7518c5864305befc478b019c0b34.dindex.zip.aes
Mar 18, 2018 12:02 PM: put duplicati-bfae9198855d44a08b602cfc999eaa8c0.dblock.zip.aes
I have seen other topics on the same subject but no resolution AFAIK. Help!
Thanks.
The deleting unwanted files process usually only runs when a lot of remote volumes needs to be deleted or when they need to be compacted.
Deletion is easy, but compacting requires downloading the volumes that are bloated with deleted files and then combining any files that are still in use into a new volume and re-uploading that.
My guess at why it’s taking so long is because you have a lot of file changes in those two weeks and a short retention policy.
You can speed it up by disabling auto compacting with --no-auto-compacting but I wouldn’t recommend it because those deleted files would not be removed from remote storage until the volume they’re in is completely deletable.
Depending on your exact needs you may benefit more from a slightly different backup plan. If it’s just about having a restore point every 2 weeks I would do more frequent backups and then have a retention policy on to remove the unwanted versions. That way each run would have less work to do.
Or you can take a look at your backup data and see if perhaps you’re backing up files that you don’t need to that change unnecessary often like temp file directories or perhaps an application/binary folder.
Actually this job is backing up 6 Virtual Machine files varying between 5G and 32G. And there were no changes compared with the previous backup. (These VM files are in MacOS called packages that are seen as directories by Duplicati?)
Uh, that would explain it. Those VM packages sometime have just a single file with all the data in. So any change in the VM even just a text file being updated may cause the entire VM to be re-uploaded and replaced on the backup destination. Which means tons of volumes are suddenly almost emtpy and need to be compacted (when a previous version is removed from the backup set)
I personally don’t think Duplicati is good for VM backups, but perhaps some people here on the forum have experience configuring it so it works better.
Thanks. It would have been nice to just set options and schedule and forget about it with Duplicati. It would be nice to hear how to configure these types of backup!
I agree - if you’re using Duplicati to back up the virtual disk FILE (essentially an entire image backup), there will likely be a LOT of add / delete activity going on with every backup. Not that it wouldn’t work - it would just be…messy.
Now if you’re running Duplicati INSIDE the VMs to back up individual files, that should work just fine.