Multiple smaller backups or 1 big backup

I need to backup several big folders from one PC, would it be preferred to make 1 backup, backing up all these folders. Or would it be wiser to split it up in more separated backup instances (each instances only having one or 2 related folders).
The reason I am asking this, I am a bit worried about corruption. If one big backup gets corrupted somehow, I loose everything. If using smaller instances, I only loose one of the instances.

1 Like

Duplicati is very robust at handling “corrupted” backups. Anything recoverable will be recovered. For best results, you should really have multiple backups of the same thing, one of which offsite.

That being said, I believe the main side effect of doing multiple backups instead of a single one is that de-duplication is less effective. Duplicati chops files up into smaller blocks and only backs up one copy of each block. Having more files in a backup increases the chance of having duplicate blocks (so needing less archive storage space).

A benefit of doing multiple backups is that you can fine tune the backup jobs settings for the content. For example, if one backup is going to be all music, photos, and videos you can likely set compression to 0 (these items are already compressed) and lower version retention counts (as these files likely don’t change much).

1 Like

and lower version retention counts

Isn’t that dangerous? I’m thinking about doing the same as the topic starter but since even those files change a lot in my situation I run a backup at least every hour but want to lower this to every 30 minutes. Putting it to let’s say 24/48 versions would mean that if there is by example a crypto attack everything could be lost in a day since the correct files are already deleted. Or am I seeing this wrong?

Currently I just use one backup set for all folders (multiple ones for multiple destinations) with compression to 1 and retention for 1 year. (Connection and storage are no issue for me but local performance is)

I don’t know that I’d call it dangerous in my case, but for you it might be. If you’re worried about a crypto attack some things you could consider include:

  • in step 5 (Options) set “Keep this number of backups” to “Until they are older than” instead of a specific number. Yes, this will cost more in archive storage, but as long as you detect the issue within that time frame you’re “safe”
  • if using version 2.0.2.10 canary or newer consider looking at the new --retention-policy advanced option which can be used to lower your version density (“thin out”) as files get older. With that you can still keep files longer, but not take so much of a hit in storage costs
  • set up multiple backups with differing frequencies as sort of a manual “retention-policy” design. Your “every minute for 60 minutes” backup might get crypto locked, but your “every hour for 24 hours” or “every day for 30 days” backups would still be viable

Note that there is also a Drastic change alerts discussion recently started which includes ideas such as alerting users if more than X% of files have changed since the last backup (such as might happen in a crypto lock attack).

Thanks for the clarification, I already saw that topic and looking forward to this. :slight_smile: I’m not using the canary build (too afraid, maybe without reason), but read about the retention policy and looks good.

Since splitting the backup for multiple file types, retention and multiple locations seems a bit hard to manage I will just stick with the current setup. As mentioned the connection and storage are no issue to I don’t bother to much about that. Thanks again.

1 Like

Hi JonMikelV,

I would actually say that I could set the retention count to unlimited for files that do not change a lot. I will most likely only have 1 version of it.
I am not a fan of lower retention counts. For files that do not change a lot, it doesnt matter anyway (family pictures, scanned invoices etc) since you already have a low amount of versions (typical 1 version). And for files that change a lot (word documents etc) it is often important to keep all history (this new retention policy is very interesting for this).

Nor am I - the peace of mind I get knowing I can go back to previous versions is worth the small cost of disk space. But every user case is different so I thought I’d point out the option. :slight_smile:

And the retention policy feature was of great interest to me as well (which is why I threw some money at the Bounty for it). :money_mouth_face:
https://www.bountysource.com/issues/38912971-retention-policy-like-time-machine?utm_campaign=plugin&utm_content=tracker%2F4870652&utm_medium=issues&utm_source=github

Sorry for bumping up an old topic but -

Just recently started using Duplicati with B2 storage.
I would like to backup ~100 GB to B2 from a single machine.
This includes multiple local directories spread all over the local filesystem.
The contents are varied - documents, text files, images, audio / video files, Thunderbird mail storage, etc.

What is the recommended backup strategy here ?
Currently, I have created one bucket in B2 with a folder_name and all currently selected local files / directories are being backed up to that.
I have not yet selected all my local files for backup since I am still testing things out.
Should I continue backing up everything to that single remote folder ?
Or should I create a separate remote folder for larger data like images / audio / video ?

My initial idea was to just have one remote folder and backup all ~100 GB to that location. Is this correct ?

He @gotunandan, welcome to the forum!

Top me, correct is what works in your situation. :slight_smile:

The benefits of a single big backup are simplicity (no cross scheduling issues) and potential deduplication benefits resulting in less destination (B2) storage being needed.

The benefits of multiple smaller backups are that you can have different settings and schedules for each backup. For example, if your audio / video files rarely change then they might only need to run once a week and you might only need to keep 2 versions of each file. Then you’re free to back up your documents multiple times a day and keep LOTS of backups.

On top of that, you can thin out your versions (with the --retention-policy parameter) at different rates. For example, kill versions of Thunderbird and text images FASTER than documents…

I know I’m making it sound like multiple backups is “better”, but it really is up to you. With ~100GB of source files I doubt you’ll run into too many performance issues with a single backup (unless there are a LOT of small files).

Just remember that if you change your mind later you can’t “merge” backup histories (versions), but you can move source folders from one backup set to another and keep the old backup set around un-scheduled in case you need to do a “legacy” restore.