Sizes for ~20TB of large files (1gb-10gb each)?

Support
1

I’ve been using Duplicati on a Windows machine for a couple years but only doing backups of small files (1TB worth of document type files and photos). Now I’ve setup a new computer with a lot more storage and want to start backup up my NAS which currently has around 20TB of large files (media) that are between 1gb and 10gb each (small percent larger, equal number of smaller files with metadata that go with each media file).

Currently I setup, and ran successfully, a backup using blocksize 50mb, dblock-size 5gb. I set “remote volume” size to 5gb as well as I believe this is the same as dblock-size but it gave a warning (but still let me do it). Note that the backups are all on my local LAN, both NAS and backup box have 10gbe. These files are not very compressable as source and backup are same size but they are x264 and x265 encoded data mostly. Probably less than 10gb if you add all the small metadata files up (one or two for each larger file).

My question is this: Does my blocksize and dblock-size seem reasonable? Is remote volume and dblock-size the same? My goal is to not end up with 20TB being 20,000 files in one folder, so I went with 5gb each which means it’ll be 4,000 files which still seems a bit excessive. Currently I’m backing up individual sections the largest was 2TB but I now have one section that is 10TB and it’ll take a day to back up so I’d like to get it right the first time if possible. Any suggestions appreciated.

Ohh, lastly since these files are large media files they do not change often. I’m not too worried about a few blocks needing to be redone if a file is changed since that’ll be infrequent. I understand restore will take longer, but I don’t plan to restore much unless there’s a total failure in which case I’ll be restoring it all. I mean I’d hate a single file restore to take 5 hours, but this is on a local machine which is otherwise performant. Any tips appreciated!

2

Yes, they are the same, and the UI warns about such a size, because it is most likely not a great size to use for transfers over the internet. For local transfers, it depends on the setup, but it sounds like you understand the tradeoffs.

Yes. The dblock-size is the name of the CLI parameter and refers to the file extension .dblock. The name “Remote volume” was chosen to be less technical and convey the general idea of storing data in volumes.

It depends on your setup really.

The blocksize can be large, which will result in fewer blocks and give better performance on database queries. The downside is that changes less than a block are treated as the full block has changed. For encoded video (+images, +compressed files, and more) this is usually not relevant as a single changed bit will cause a cascade of changes in the file data, and remove any possibility of data remaining the same.

The only place where it could have an impact would be editing file metadata (in the header or footer). Here, the changed information would be a very small block, but since 50mb blocks are processed, this will result in a full block being seen as changed. For most cases, I would not expect this to happen often.

For the dblock-size, the tradeoff is mostly for restores, as any given block requires the full volume to be transferred before the block can be extracted. If your data changes a lot and is spread across many large volumes, there will be significant IO overhead to restore a file. But if the data is mostly static, all parts will be in the same volumes, so you would at most download one extra volume if you restore only a single file.