Restore stopped with database or disk is full database or disk is full

[carljohnsen]

@Nosenko

I am restoring to a hardware RAID-6 drive using an external USB drive for backup.

That sounds like there should be plenty of capacity for the restore process. Do you see any bottlenecks on the target drive when its being run with 12 writing processes?

@Nosenko

You made some poor decisions as to use cases then and configurations that are not the default. I was not aware of the 50 MB volume size, but I started my backups a few months ago at 500 MB, then once the baseline data was stored in these mostly unchanging volumes, I switched to 200 MB. That appears to collide with the new restore’s logic quite noticeably.

I can see that’s true for your setup. I would argue that it’s not as much that the new restore flow doesn’t support larger volumes, but rather that a re-download of them are more expensive. With the current default cache size setting, I assumed that 100 downloaded volumes would be sufficient for most backups. However, looks like it’s not uncommon that files are spread across more than 100 volumes, leading to the cache issues. Restoring multiple files concurrently further stress this cache boundary, causing them to compete for resources. Changing configuration parameters is always a trade-off: if there wasn’t a trade-off the default would (or at the very least should) be the optimal value.

@Nosenko

When I re-ran the restores with smaller requests (roughly 1 and then up to 2 TB), no errors were thrown and it seemed like it ran normally. It really increased my babysitting time and effort, trying to get work done while I peered at the logs so I would at least know if/when things were going bad.

Was this with the new or the legacy restore flow? If you try the new restore flow with --restore-file-processors set low as well, then it shouldn’t be requesting too many volumes at once (assuming that blocks haven’t been assigned to more than 100 volumes in a round-robin fashion). For the legacy flow, the issue is likely related to the SQLite error mentioned earlier.

@Nosenko

The new restore is just not working right with the two bugs I have found, and perhaps this third one you admitted to. If someone doesn’t modify the --restore-volume-cache-hint= setting, they may be toast if their volumes are larger than expected. I think you should immediately re-think and rewrite the code that makes an assumption as to volume sizes.

The cache size configuration has been addressed in this PR where the default becomes “use as much cache as possible up until there’s only 1 GB left, then perform premature eviction”. It will take some time to make it to a release. Regarding changing the default concurrency parameters, I don’t have a good “new default” yet.

@Nosenko

There are many resources about Duplicati, which I have been using for most of a decade, where volume sizes are discussed and suggestions are made to change them.

I understand that this information exists, but it needs updating (some of which I cannot do myself) to reflect the characteristics of the new restore flow. Larger volumes still help with backends that favor fewer but larger requests. This can also hold true for the new restore flow, given that there’s enough space on disk to hold the volumes to allow for parallelism. If there’s no premature eviction, then the only downside to larger volumes is that each volume can only be accessed sequentially. Thus smaller volumes allow for smaller resource locks, in turn increasing the parallel capabilities.

@Nosenko

But both the new and the old restores seem to struggle a bit. I tried tweaking the new method and it failed due to the bug I reported about thrashing, and then failed due to another one.

I think the primary issue for the new restore flow is the cache issue. As noted, the new default should help with this and we’re still working on improving the cache management further. The legacy restore flow breaks due to the SQLite related issue.

@Nosenko

I wanted to keep the number of files low on my external drive to improve efficiency, but I had no idea it would make restores crawl or halt.

Right, makes sense given your external drive. The new restore flow can be tuned to work with this, as you’ve already found with setting --restore-volume-downloaders=1 ensuring that only one volume is downloaded at a time. However, with the cache issue one volume is fetched multiple times, which I assume is causing the stalling.

@Nosenko

The old method got me back up and running and finished the restore after several days, but there are so many problems with Duplicati that I must stop using it. Problems start with the counter-intuitive nature of the web interface and words that need help explaining… but there is never any help text for. Obscure settings must be dug up to fix what would be easier for the user with a Q&A and recommended settings, guidance, menus, helps, reminders, suggestions, etc.

I understand your frustration, and I’m sorry to hear that you’re considering stopping using Duplicati. I noticed that you’re using the old web interface: is it still an issue with the new web interface? Regarding information in general, the docs cover a lot of the settings and use cases. Are there any particular topic missing from these that you think would be helpful to add? Contributions are also welcome there.

@Nosenko

I think it’s been 9 years or so for me. It never got polished. Never got easier. And the Windows underwhelming versions of backup, although not very smart, are robust enough and a magnitude of precious time faster to restore when problems arise.

Things that could be parallel aren’t, such as the restore that serially looks at my hard drive and determines what is (and what isn’t) already on the drive and whether it matches the one on backup. Once it obtains a list, it could go into a FIFO buffer and begin the restore in a separate thread. This will simultaneously read and write to that disk, but some drives (SSD and some RAID drives) can perform this without much of a performance hit.

This is exactly what I was trying to achieve with the new restore flow: parallization of the restore process, which is described in detail in the corresponding blog post. Instead of serially looking at each volume and scattering the containing blocks across the target disk, the new restore flow starts by parallelizing on the file-level: each file is restored in parallel. This also ensures sequential write (assuming the whole file needs to be restored) which benefits any disk.

The flow starts with the FileLister communicating the files to restore to the FileProcessors (of which there are multiple running in parallel). For each file, the FileProcessor checks whether the target file exists (e.g. for picking up on a previous restore), whether it has the correct size and hash, and if not, finds which blocks are missing. Each missing block is then requested from the block cache. If the block is there, it’s returned, if not, it asks the volume cache for the missing block. The volume cache then checks if the volume is already downloaded, and if it is, it sends the volume to the VolumeDecompressors (of which there are multiple running in parallel) which extract the blocks and send them to the block cache. If the volume is not downloaded, it sends the request to the VolumeDownloaders (of which there are multiple running in parallel) which download the volume, send it to the VolumeDecryptors (of which there are multiple running in parallel) which decrypts the volume and sends it back to the volume cache. The volume cache then sends the volume to the VolumeDecompressors which extract the blocks and send them to the block cache. The block cache then sends the blocks back to the FileProcessor which writes them to disk. So the new design has a lot of parallelism, concurrency, pipelining, caching, and asynchrony built in. Each communication step is a FIFO queue allowing for burst requests. The network is tunable (--restore-file-processors=, --restore-volume-downloaders=, --restore-volume-decryptors=, --restore-volume-decompressors=) allowing for more or less parallelism depending on the bottleneck. The cache is tunable (--restore-volume-cache-hint=, --restore-cache-max=) allowing for more or less caching depending on the disk space and RAM available.

Given the complexity and immaturity of the new restore flow, and due to it having lighter resource requirements because of its serial nature, the legacy restore flow has been kept.

@Nosenko

But here I am asking for a feature or bug repair again. That’s not the hallmark of a mature, highly efficient program that has all of these shortcomings. Why haven’t the developers thought about how to more efficiently copy a file? Wasn’t that in programming class 201?

There is no need to ridicule me or the other contributors.

@Nosenko

And if you do things like that hammering Google Drive or Dropbox with 12 simultaneous file requests, why not do it on the local machine?

I’m unsure of what you mean: both the new and legacy restore flows treat local and remote backends the same, trying to maximize work effort. What’s missing on the local machine?

@Nosenko

Oh, one more bug:

The log results claim that 0 files were restored, no matter if there was 2 or 1,000,000 files. The warnings are always inflated. And the only thing I trust is maybe the number of errors. Here’s an example … I can’t remember how many files got restored, but it was a lot…

That’s definitely a bug, and I haven’t seen it before. Is this occuring in the legacy restore flow? Does the log file reflect the same counts? Does the new web interface show the same behavior? Are all of the warnings the same and what are they? When I run a backup and restore with 2.2.0.3 from a local folder to a local folder I see the correct results under the log, both for the old and new web interface.