Patching files with local blocks

Support
1

Howdy,

I’m trying to restore a set of files on Duplicati 2.0.3.3_beta, the restore seems to have hung with the status “Patching files with local blocks …”. I’m seeing a bit of CPU usage from Duplicati but no other action for many hours (it initially ran for over 18 hours).

I have “–patch-with-local-blocks=false” set in my Default options, so I assume that these would be carried over to the restore process?

Any idea how to determine what it is doing, or how to otherwise convince it to restore the files?

2

I just checked in, I’m still waiting at this same status 17 hours later.

3

I’m not sure backup parameters are carried over to restores - even if you use the in-job Restore feature.

I’m also not entirely sure of the difference between --patch-with-local-blocks and --no-local-blocks.

If it’s an option, if suggest restarting the restore with both those options manually added in the restore GUI.

--patch-with-local-blocks = false
Enable this option to look into other files on this machine to find existing blocks. This is a fairly slow operation but can limit the size of downloads.

--no-local-blocks = false
Duplicati will attempt to use data from source files to minimize the amount of downloaded data. Use this option to skip this optimization and only use remote data.

4

Even if not, the default should be false. After 36 hours or so I set up a fresh install of Duplicati on a fresh VM with relatively limited bandwidth and CPU and was able to restore the needed files in under 30 minutes, so at least the immediate fire is put out.

5

I think it is, but I think the code treats it as true if present at all, even if set to false. Seemingly a code bug:

Compare the above to this, which is what I expected to see, using ParseBoolOption, not ContainsKey:

Intended boolean option behavior is described here and here. Default is false, but mentioning the option is implicit true unless an explicit false is given (as you did). Problem is the code didn’t use ParseBoolOption.

This might be what you’re seeing, although you’d have to delete that option from defaults to know for sure:

As to what the options do, a given option has a short description and a long one that’s usually a superset. Here though, the long omits some critical applicability details. Command line help has both, and defaults:

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe help no-local-blocks
  --no-local-blocks (Boolean): Do not use local data
    Duplicati will attempt to use data from source files to minimize the
    amount of downloaded data. Use this option to skip this optimization and
    only use remote data.
    * default value: false

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe help patch-with-local-blocks
  --patch-with-local-blocks (Boolean): Use local file data when restoring
    Enable this option to look into other files on this machine to find
    existing blocks. This is a fairly slow operation but can limit the size
    of downloads.
    * default value: false

C:\Program Files\Duplicati 2>

So the short description for --patch-with-local-blocks is the only one that explicitly says that it’s for restore.

They both attempt to avoid getting remote blocks. One is limited, while yours sounds big. Was disk busy?

Code sequence here is:

  1.             //Look for existing blocks in the original source files only

  2.             // If other local files already have the blocks we want, we use them instead of downloading

  3.             // Fill BLOCKS with remote sources

and the restore was probably churning at #2. #1 is --no-local-blocks, default false. Double-negative = true. While beneficial, this can actually get in the way of proving a restore works (because it uses local blocks).

6

The disk was largely idle most of the time, although every few hours there’s a 45 minute process that pegs the disk (waiting 6 hours before it starts again).

I was watching Process Monitor, there was only CPU usage from Duplicati and virtually no disk or network access. Disk queue overall was sitting at 0 - 0.1 for the majority of the time.

The subdirectory I was restoring into was virtually empty, after about 30 hours a bunch of 0 byte files appeared and then nothing again for the hours until I decided to build a VM just to handle the restore (and completed the entire restore on this VM before killing off Duplicati on the original machine).

The parsing described in the code explains why it was trying to work with local blocks at all, thanks, but even so I don’t really comprehend why nothing happened, Duplicati had time to scan the entire drive several times over if it wanted to do so.

I don’t believe the machine was particularly overloaded in any fashion both because I keep half an eye on it all the time, and because after I set up Duplicati on the VM I got Restic installed, configured and completed over 50% of a backup (uploading directly to B2) before my VM finished the restore, all the while the original Duplicati install was still chewing away without a single actual file being restored.