Errors during restore

Hi all, I would like to ask for your help, please. My 4TB drive had a failure and I need to recover all of our pictures, videos - from wedding to our kids etc. This was “D” drive - separate from the OS drive that has Duplicati installed.

I had setup a remote backup to a laptop with SFTP at a friends house with 10 TB external drive connected.

I brought it home so it’s on the same local network. I have changed the DNS so it points to the local IP and now Duplicati is able to run. However it only restores the folder structure, not actual files!
The status bar says: Scanning for local blocks and in the Live log I have setup I see this - seems it’s there for each file.

  • Sep 27, 2020 10:19 AM: Target file is not patched any local data: e:\duplicati-data_scans\scuba-home page.pdf

  • Sep 27, 2020 10:19 AM: Local source file not found: D:_scans\scuba-home page.pdf

  • Sep 27, 2020 10:17 AM: Target file is not patched any local data: e:\duplicati-data_scans\scan0088.pdf

  • Sep 27, 2020 10:17 AM: Local source file not found: D:_scans\scan0088.pdf

Can anyone please point me to what I need to do to restore the data?

running Duplicati - 2.0.5.1_beta_2020-01-18

I’d recommend using the –no-local-blocks option so it skips trying to look for local blocks, which it won’t find anyway if your original source data was lost.

Is the restore still in progress? (Does the Live Log say Duplicati is doing something?)

Yes, restore in progress, live logs keep the same 2 messages for each file:

  • Sep 27, 2020 12:42 PM: Target file is not patched any local data: e:\duplicati-data_scans\vetrovsky - offer letter MI.pdf

  • Sep 27, 2020 12:42 PM: Local source file not found: D:_scans\vetrovsky - offer letter MI.pdf

EDIT; and it seems I can’t stop it for now, tried both stop after the current file and stop now and it keeps going:

  • Sep 27, 2020 2:19 PM: Target file is not patched any local data: e:\duplicati-data_scrapebox\Blacklist.txt

  • Sep 27, 2020 2:19 PM: Local source file not found: D:_scrapebox\Blacklist.txt

  • Sep 27, 2020 2:18 PM: Cancellation Requested

Welcome to the forum @tomasv

From original post, is below a typical timing? It seems long. What’s at D: (drive that failed) now?

I would have thought it would race right through these and find nothing quickly, and then move on.
Technically there are no errors here. You probably have live log at Verbose level or more detailed.
Eventually it should finish seeing if it has any source files, and go to remote. Question is timing…

D: is the old drive that failed - it is still connected as I am running Stella Data Recovery in hope to get some additional files that are not on the backup.

Should I shutdown the PC. disconnect the old drive, do recovery from backup and only after that try to get some of the old files back?

Assuming that’s Stellar Data Recovery, web site indicates it’s for certain kinds of errors but not others.
Any idea how your drive failed? I/O errors would probably cause some slow operations, if the firmware attempts to read sectors that are giving errors. There are some recovery tools that take this approach. Other sorts of errors might only be filesystem errors, possibly even fixable by fixing the NTFS formats.

Are you able and allowed to read the drive while Stellar is running , e.g. to copy a file Duplicati named? Possibly that will be slow (or fail, or both), and Duplicati without no-local-blocks will also attempt reads.

I’m still asking for timing information. If per-file time looks really slow, it might be due to a slow D: drive.
If times vary, and some are very fast, then perhaps you can stand to let it finish seeing what it can find.
Scanning the failed drive for data blocks for the file restore is sort of a questionable proposition though.

Does Stellar comment on that? I don’t know exactly how it works, where it’s putting results, changes to drive, progress rate, etc. From a Duplicati point of view, you should probably try copying job’s database (see path in Database screen) before doing any sort of hard Duplicati stop, to have a second copy of it. Although in theory it can be rebuilt from backup files, that can take awhile, especially for a large backup.

Where is Duplicati restoring to? All I heard of is an OS drive, the failed 4 TB drive, and the backup drive.

Thanks, correct observations - I added a new E:\ 6TB drive that I am using to restore the data so the failed one is read only. After your suggestions above I assigned it a different letter in the OS and the restore ran OK, until about 50%. So thanks for the tips!

Now I see:

File hash is LpOTCLBVnjhM46RyqgAdEvuH+HXXvSutoGcdbd/jGIU=, expected hash is qEcOmSwVtZgXOt/e83tBehKqToSGmHu3yHhHhunuc8w=

But the backups were completed without errors, not sure how to fix that.

Abbreviated message? Does it fit format below? How many were there? I think this is a file which didn’t restore to its original form somehow. At backup, a mathematical hash is taken of the file’s data content.

Above restore checking looks like it should happen after the restore, so I’m not sure how 50% factors in. Where is the number from? For the file that didn’t restore, what is its exact size, and is it something you could sanity-check somehow, e.g. by opening it? This error has a few reports, but no clear conclusions.

EDIT:

I guess I should ask if any other restore errors happened. You’re seeing the final test, but an earlier error could help explain how the file wound up looking wrong. Especially look for any errors on “patch” failures.
Duplicati assembles restored files a block at a time, and “patch” refers to putting a block into some file…

Yes, found few patch error as well. I don’t see them at this moment as I started the next restore job attempt.

The files are large - about 4.5GB each and are raw video files. One of them is definitely corrupted, the other looks OK but it will require actual play to verify.

So if I understand correctly the patch might failed because of number of reasons - from network error to backup consistency. I may try to restore just those 2 files next, without running the whole restore job, which in this case was about 480 GBs.

EDIT:
Now I am seeing errors on almost every file:

  • Sep 28, 2020 6:20 PM: Target file is not patched any local data: e:\duplicati-photos_New2020\Leavenworth - July\20200711_143858.jpg

  • Sep 28, 2020 6:20 PM: Local source file not found: D:\Photos_New2020\Leavenworth - July\20200711_143858.jpg

  • Sep 28, 2020 6:20 PM: Target file is not patched any local data: e:\duplicati-photos_New2020\Leavenworth - July\20200711_143631.mp4

  • Sep 28, 2020 6:20 PM: Local source file not found: D:\Photos_New2020\Leavenworth - July\20200711_143631.mp4

Do you still have no-local-blocks checkmarked in options, either per job options or global Settings?
That’s what was suipposed to keep it from looking for the original file, except now it will fail faster…

Good plan. Ability to read the backup has seemingly been related, however your drive is local I think.

Is the log you look at able to expand them by clicking on the error line to see the reason for problem?
If you get a whole lot of these, log-file=<path> and log-file-log-level=warning (or more) may be easier.

Not much progress - I was able to restore one of those 2 files and I moved onto the next backup set.

It’s a large one, 3.5 TB in total The restore job ran until this point, status shows “downloading files” but the SFTP server shows no activity in the last 24 hours and no files have been restore (again, only the directory structure). I am assuming I need to start all over again.

BTW thanks for your help - I owe you beers!

  • Sep 29, 2020 9:05 AM: The operation List has completed

  • Sep 29, 2020 9:05 AM: The operation List has started

  • Sep 29, 2020 9:05 AM: The operation List has completed

  • Sep 29, 2020 9:05 AM: The operation List has started

  • Sep 29, 2020 9:05 AM: The operation List has completed

  • Sep 29, 2020 9:05 AM: The operation List has started

  • Sep 28, 2020 6:27 PM: Backend event: Get - Started: duplicati-bb66bd45720934988af2e2e53c035c6dd.dblock.zip.aes (4.98 MB)

  • Sep 28, 2020 6:27 PM: 27714 remote files are required to restore