I am investigating replacing Crashplan with Duplicati, but am finding that Duplicati appears to require a free disk space equivalent to at least the amount of data being backed up, for it’s temp files. My disk keep filling up with temp files and the initial backup never completes as the disk gets full…
If this is by design I urge you to seriously reconsider; it makes backing up large amounts of data with Duplicati impossible unless one has a lot of free disk space available.
Crashplan & other competitors does not have this limitation.
Ok, I found the issue, and it was my own fault.
I had set the Upload Volume Size to 500Gb, thinking this was a limit on how much space Duplicati could use on the target. Which of course it isn’t, but that is hard to know given the lack of documentation.
Resetting it back to 50Mb solved my issue, I can now see Duplicating creating 50MB data files and uploading them, deleting them as they have been uploaded.
I made the same mistake… I originally set the Upload Volume Size to something gigantic for my first test backup. Might not be a bad idea for them to add a note on that config page to make it clear what exactly the Upload Volume Size is.
what about a warning when more than 1GB (?) is selected? There could be a more detailed explanation preventing users from using this option by mistake…
I remember, though, that I was also confused about this and thought about increasing it when I first installed duplicati. Or did I actually increase it? I think I did. So I have to admit that some things that seem obvious to some are not obvious to others…
For many people, the word “Volume” is synonym for “the complete disk”. Add “Upload” to it, then “Upload Volume size” is easily interpreted as “the total backend storage quota”.
If there’s confusion about this term, why not rename it? What about Block Container File?
The Upload Volume size (AKA dblock files, default 50 MB) can be changed, the block size (file fragments that are stored in the dblock files, default 100 KB) cannot be changed agter the first backup has been made.
Note that the already uploaded dblock files keep the same size. So if you choosed an initial size of 10 GB, make the first backup and change the size afterwards, you have to download 10 GB when restoring a small file that is uploaded during the first backup operation.
Well, after playing with this a bit I found that my backups appeared to run faster when I set the value significantly higher than the default 50MB.
I currently have it set to 5Gb, which I think is reasonable enough given the average size of most of the files I want to back up (3-5Gb, pictures).
Imho I believe 50MB to be much to conservative, nothing is that small these days.
Does the Upload Volume Size have any impact on compression efficiency btw?
A larger Upload Volume size will decrease the number of files at the remote side. A lage number of files in one folder can decrease the performance of the backend server.
Restoring files will take more time when choosing a large Upload Volume size. I guess 3-5GB is the total size of the pictures you want to backup, not the size for each picture.
Let’s assume the avarage file size of a picture is about 4 MB. With an Upload File size of 5 GB about 1000 pictures will fit in a single volume. If you want to restore a single picture, Duplicati has to download the complete volume (5 GB) to rebuild the Original picture of 4 MB. If something has changed and the needed blocks for the picture file are devided over 2 archives, even 10 GB or more has to be downloaded for restoring a single picture.
Not much, if any. All volumes are standard Zip files. Maybe there’s a little more overhead when writing to many small Zip files, but I guess this will not be a big difference.
BTW: A good way of showing your appreciation for a post is to like it: just press the button under the post.
If you asked the original question, you can also mark an answer as the accepted answer which solved your problem using the tick-box button you see under each reply.
All of this also helps the forum software distinguish interesting from less interesting posts when compiling summary emails.
I have an “unreliable” network connection, so I set this to 10Mb. Maybe an option (or default) to use automatic sub-folders for files would help. Adding a sub-folder with first char would reduce 1/34 the files, and using two subfolders 1/1156 (I also use backuppc and it uses 3 sub-levels to reduce files in one folder).
This could help for large lists of files at the backend, but I can think of some backdraws:
Implementing this structure in a new version would make existing backups unusable, unless both file structures will be supported, which will make the software more complex.
This would add a new requirement to the very limited requirements for backends (PUT, GET, LIST, DELETE). Some backends could potentionally not be able to create folders, which would kick them out as backend for Duplicati.
Internal administration would be more complex, which could slowdown backup/restore operations.
I don’t know how difficult it is to implement, so if the issues above can be addressed, it could help with large backups.
Unless I’m thinking incorrectly about how this all works, it might also be worth having a warning on the larger sizes that at least DOUBLE the selected size might be needed in free disk space to do a restore. So if 500M was selected, then when doing a restore of a 750M file will require 500M the download of the first dblock then another 500M for the decompressed contents, then ANOTHER 500M for the next dblock, and finally 250M for the remaining decompressed contents.
So in that abusive example we actually needed 1,250M free space to restore a single 750M file from a 500M dblock size.
button under the post.