A “curiosity” question that’s hopefully relevant enough to include in this thread:
Does the upload volume size impact how much is downloaded when a file is downloaded for archive verification? (I hope that made sense…)… another way to put it, if I have the upload volume size set to 50mb, then does that mean that 50 mb is downloaded for each verification? If I set the volume size to 750MB, does each verification download 750MB?
Also, when restoring a single file, is duplicati able to just download the file, or does it have to download the entire 50mb (or whatever size it is) volume(s) to then extract the file from there?
Mostly I’m just trying to understand how it works so I can come up with the volume size that makes most sense for the amount of bandwidth I have.
As a side note…after playing with duplicati for a few weeks, I’m very, very impressed with the whole thing.
Tyson_Bryner, it might be easier when asking multiple questions so use bullets or a numbered list. Hopefully I got all of your questions covered:
Yes - whatever the upload volume size is set to is how much will have to be downloaded for a single verification to occur
Yes - the entire archive file must be downloaded so that a single file can be restored. But it’s not quite that simple as Duplicati actually chops up your files into smaller blocks and stores each of them in the archive (I believe this is how de-duplication is managed, at the block level) so ALL the archive files holding ANY of the blocks of your actual file-to-be-restored will need to be downloaded. So if you have a volume size of 50M, and a 100k block size then your theoretical 1 Meg data file might actually be stored in the destination in anwhere from 1 to 10 (1 Meg / 100k = 10) different archive files (ALL of which needing to be downloaded for restoration).
Gotcha… That’s kind of how I imagined it all working. So I figure it makes more sense to stick with volumes of smaller sizes in most cases. If I’m storing a Terabyte or two of data, it’s going to make a huge number of archive files, can anyone think of any downside to having such a large number of archive files in your destination?
Some filesystems don’t like lots of files, and I think some cloud providers might have some limitations on such things.
Your best bet is probably to check out the Choosing Sizes in Duplicati page and see if it gives you any ideas (sorry, I should have linked to that earlier like kendkendk did above):
I’ve had failing backups for the last few days with my disk filling to 100%. Solve now thanks to this discussions. Come on guys, someone create a user manual for this software. It looks pretty good but people can’t guess how to use it!
By any chance are you building this in something like a google doc that I or others could have “suggestion”-level access to? I’d be happy to go over it and make suggestions… i.e.
For now, I use a local MS Word install, because I’m used to it, so it works the fastest for me.
Once complete, I’ll find out what’s the most convenient way to share.
(btw the hick-ups spelling error is fixed, text was pasted from here, so that should be updated too).
I ran into this too. I set it to 1TB and after 30 minutes my C drive filled up and I had to track it down to a 50GB temporary file. I think the current wording (“Upload Volume Size”) is exceedingly misleading since:
It’s next to the “Keep this number of backups” option, so I assumed it was about retention.
I can’t think of a reason my backup program would care about the source volume (and it could easily just detect this automatically).
The option doesn’t actually describe what it is for. I think “Upload Chunk Size” or “Backup Chunk Size” or something would be more appropriate.
In retrospect I’m going to guess that “volume” is the internal term that Duplicati uses to describe backup chunks, but I don’t think that’s self-obvious and I didn’t see it referenced elsewhere in the UI so this is probably a case of leaking implementation details.
I would recommend removing this option from the UI entirely. It seems unlikely most users should need to configure it. If they do, they could do it as an advanced option.
If I’m overlooking something and it’s actually worth making this a top-level option that every user must consider changing, then perhaps rename it to “Upload Chunk Size.” I’d also strongly advocate for a tooltip or other mechanism of adding descriptions to options so that you can document the trade-offs of changing the option.