Newbie question on a large backup

Support
1

Completely new to Duplicati, At the moment I have setup a backup to a networked drive on windows server 2022. The backup is 3.5TB large and obviously took a number of days to complete. It has now complete (status bar says (0 bytes) to go) but from the home screen looks to be stuck. I know it needs to run a deduplication on the backup and because I encrypted the backup it has stored everything locally and assume that it will then move the encrypted files to the network drive once complete. my question is, the backup has been stuck at 100% complete for a couple of days however in the live log and under “ËxplicitOnly” i can see multiple instances of

10 Apr 2024 11:43: Starting - ExecuteNonQuery: UPDATE "Block" SET "VolumeID" = 8 WHERE "Hash" = "9m61VcTDVT+p3v4Y4znT5qNcNBZOek4g7OGZ4HrSuCw=" AND "Size" = 102400 AND "VolumeID" = 11

It is also holding about 10gb of RAM (not an issue as the server has 64GB to use) so even though the backup looks stuck it is obviously doing something. I am just wondering if anyone can enlighten me on what it might be doing and how long i can expect to wait for it.

Small side note i also set the “Remote volume size” to be 5TB, i did this at the time as i misunderstood what it was doing but now realize this was probably a mistake, can i change this after/if the backup completes or am i stuck with this size now (its only going to a networked drive, not going to an offsite backup yet)

2

Welcome to the forum @D4NNK

Can you post status? Left numbers count up as files are found, then down as files are processed. Upload to destination starts sometime, and numbers on the right show its average speed. There is text status as well.

That’s actually a big deal, and it might be related to the slowdown.

It moves them as each remote volume (default 50 MB) fills, otherwise it could hog excessive local space, making a huge slow upload at the end. Maybe you’re in one now. What’s visible at the remote? Anything?

Sometimes Explorer is not very good at self-refresh. File size can be checked with right-click Properties.
Duplicati About → System info will also show you where you are in the uploading, if that’s where you are.

Deduplication happens even earlier, more or less as files are read, and before even local storage of data.

Presumably with a different Hash each time, but maybe the same VolumeID? If so, it’s probably updating database tables to show the final home of each block in some very large file that will go to the destination.

You can reduce the enormous Remote volume size at any time, but it only applies to any future volumes. Restores will be painfully slow, as the backup-in-a-volume will likely have to be fetched for almost anything.

One thing that can only be set at start of new backup is blocksize which defaults to 100 KB and is OK for backups of about 100 GB before work gets slow due to too many blocks (rough estimate, result may vary).

Does backup have to be 3.5 TB, or is there an easy way to break it apart? Number of files also matters, as files add around 2 blocks each (minimum), and rough rule of thumb says to limit backup to 1 million blocks.

If you don’t want to do the math, you could increase blocksize to 5 MB for that backup, and see how it does.

3

Hi ts678,

So i am sure it has finished the initial backup as the green bar is complete and the number of files 99088 hasnt changed in days and also shows “0 Bytes to go”

I assumue as i have set each remote volume to be 5TB thats why there is nothing on the network drive because it is still encrypting that 5TB block.

Yes as you mention there is a different hash everytime.

I am suspecting my best course of action is to stop the backup. Delete the dup-random sting of charactes files from the temp folder in appdata on the C: as this is where the files seem to be kept locally and start a new backup but this time with more reasonably sized remote volume sizes. I also think i will break the job into two jobs backing up the most crucial folders first.

I’m assuming there aren’t any known issues with stopping a Duplicati job if you intend to scrap the backup files afterwards anyways?

Thanks for your reply and future help :slight_smile:

4

Fresh start (delete the database too) should do fine.

5

Thanks for your help.

Here we go again :slight_smile: