Delete backup job before completion

[kenkendk]

Hi @Hassan_Ghaseminia welcome to the forum.

@Hassan_Ghaseminia

I don’t know what the reason is. Usually, it doesn’t give an error, but the process is very slow, taking about two to three days to make a backup. My question is, is this tool suitable for backing up such a large volume of data?

Duplicati tracks data in the size of “blocks”. The default block size is 100KiB for v2.0.8.1 and earlier. Using 100KiB blocks means Duplicati needs to keep track of approx 10 million blocks pr. TiB, and perhaps double that to account for file metadata. You can set the blocksize using --blocksize=1mb, but you cannot change the blocksize of a backup that has already created remote files. Depending on your data patterns, you can consider using a 10MiB blocksize, or larger to reduce the number of blocks.

See the article on sizes for more general guidance.

@Hassan_Ghaseminia

I’ve set up the service in Docker, and everything seems to be working fine, but the job gets deleted, and no backup is taken. When I start the job, it creates the files, and the process begins, but after a day, when it’s still in running mode, I notice that there is no backup, and the job has been deleted.

Could anyone help me understand what’s going wrong and how to fix it?

The “the job gets deleted” part sounds like Docker is restaring and you have not set up a data folder. Duplicati will use the data folder to store the database with your backup configurations, as the Docker volumes are wiped on updates and restarts. I do not understand why it would restart, but maybe it is related to the “out-of-disk” issue you mention.

@Hassan_Ghaseminia

Another problem is that the server hard drive, where Duplicati is installed, has 150 GB of free space. When I start a job, this free space gets filled up. The backup destination is on another server, so why does the hard drive where Duplicati is running get filled up?

The question is, why does Duplicati use so much cache? How can I transfer the cache volume to the backup server so that it doesn’t take up space on the local drive?

Duplicati stores a database that serves as a lookup for the remote destination. This database makes it faster to perform various operations but takes up some space. The primary space taken is from the hashes and file paths, with some additional cache space for parallel processing.

You can reduce the database size by adjusting the blocksize as described. I am not sure what remote volume size you are using, but here Duplicati will build multiple remote files (default 4) while creating the output. If you have a large remote file size, this will affect the local storage space used. You can toggle this part with --asynchronous-upload-limit=4.