Error recovery and recovery questions

1

Apologies if this information is somewhere else, but how does duplicati handle the following scenarios:

My first backup is taking days, what if the internet drops out temporarily during a backup for any reason, is it smart enough to make sure that the final backup is valid?

The last step which is apparently to upload a list of every file takes about 3 days on its own. What if that gets interrupted?

Once my initial backup is complete and I’m only adding files, will the final file list still take days to upload?

If I need to do a restore and I need the data soon, would it be possible to perform the restore from a different location such as a hard drive that was mailed to me with the data?

Thanks

2

You can definitely restore from a drive mailed to you with the data, provided the backup is working.

Application code receives or experiences various things when it comes to the first two questions. So as long as all those things are taken into account then it can successfully know what to do to keep the train on the tracks. In my experience with testing these things, it does well there, but it does have code for other services that I haven’t tested.

If your backup is too large, its going to take a while on all things, as Duplicati uses containers and for other reasons. You can do various things to affect the time to make it better or worse. Personally, I wouldn’t make such a large single backup and would split it into roughly max of 100-130GB sizes each and have multiple going and any large files that never change I wouldn’t have that continually running through automatic backups. What you do here though is up to you.

Also, having narrowed down some programming issues in Duplicati myself, I wouldn’t rely only on Duplicati if that’s what you’re doing. You shouldn’t rely only on one backup application anyway and should always use at least two different ones. But, take the risk you want.

3

Welcome to the forum @charlespick

It’s supposed to just pick up where it left off and finish backup. If your internet tends to drop, there are number-of-retries and retry-delay to configure in an attempt to keep retrying until your internet returns.

This is unlikely. It’s not that big, and is prepared before upload.
How do you know how long it takes, if it hasn’t been done yet?

You can see upload times in an information level log, e.g. About → Show log → Live → Information
The last file upload to begin (though maybe not the last to finish) is a file with dlist and date in name.

Here’s my dlist upload. Yours is almost certainly a different size. Job log says I have 6874 source files:

2021-11-11 12:53:25 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20211111T175000Z.dlist.zip.aes (994.92 KB)
2021-11-11 12:53:33 -05 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 994.92 KB in 00:00:07.6602037, 129.88 KB/s
2021-11-11 12:53:33 -05 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20211111T175000Z.dlist.zip.aes (994.92 KB)

Unless backup got lots more files, dlist upload time (whatever it is) should be similar to past backup.
Every backup gets the full file list. What varies is the amount of changed data found in the source files.

Adding files to the initial backup is actually a good strategy (but too late if you’re already doing full initial) because it lets you prioritize what to backup first, and also gets a dlist file up. Until the dlist is up, a local disaster that destroys the local database would leave you without information on the backup work.

Restoring files if your Duplicati installation is lost explains how to Direct restore from backup files however this involves recreating a temporary database from destination files, and it requires a dlist file.

Do you know roughly the size or file count of your backup source is. Multiple days uploading file contents over a slow link (is yours slow?) would be quite possible. If you need to stop intentionally, use stop button.

Duplicati gets slow on some operations like Database recreate, and Direct restore to some degree at default 100 KB blocksize with larger backups (perhaps over 200 GB) because of all the tracking of blocks.

The idea from @Xavron of having multiple backup applications is a good idea, if you want to worry less. Duplicati has gotten pretty reliable (not quite perfectly reliable), but any backup software can hit problems.

4

Thanks for the replies everyone,

In response to how do I know that the file list is taking so long to upload, I read on another thread that the final step “waiting for upload to finish” is that step. I’ve had it repeatedly hang there for days before I give up.

Today I logged in and it said 268 files to go, the exact same progress it had 2 days ago so Im not going to be going with duplicati going forward. I like the idea but it’s clearly still beta as advertised.

As for how much data I have, it’s about 4tb of macOS time machine backups stored in a sparsebundle. And I can’t easily split it up without writing separate software to do that before the backup.

Edit: I have 35mbps upload speed

5

I think this is wrong. It’s done creating backup files for upload, and draining its queue.
The dlist file is one tiny near-end file in the uploads. Check destination timestamps.

You might have to look a little further if you’re saying the number should be lower now.
About → Show log → Live → Information, or higher levels might show backup activity.
Looking on the destination is another way to see if uploads are going, and how quickly.

I don’t use macOS, but my reading says is a file holding a virtual FS for Time Machine.
If Duplicati sees it as a 4 TB file, 35 Mbps will need over 10 days to upload that big file.
After that, the count of files left to go (assuming you’re in that region) should go down.

Numbers usually start going up as files are counted, then go down as backup’s done.

On the other hand, “Waiting for upload to finish” should have limited queuing capacity,
provided you left Remote volume size alone (50 MB default) or increased only a little.
When a volume gets filled, it gets uploaded as a dblock file. There is a series of them.

EDIT:

One confusing point is “268 files to go” should have moved before “Waiting for upload”
however “repeatedly hang” sounds like there were several runs. Maybe story is mixed.
If Remote volume size was accidentally set large, one way both places could pause
is if a big file was put into a big dblock file that went into queue, then uploaded for days.