How to recover from aborted/stuck/crashed backup run?

[redusr]

@ts678

although why uploads would have trouble is unclear. Is USB-connected remote for the jobs a hard drive?

That is correct. Based on the GUI progress bar stating 0 bytes to go, I would also not expect issues in uploading. During the time in which no more activity was visible on the remote, I opened the destination directory from time to time in Windows Explorer and that worked normally.

@ts678

The stack trace isn’t that great, but maybe it’s here:

I think I’m lacking the expertise in interpreting that, but maybe someone involved in development can. Does that mean the exception occured in the write process to remote?

@ts678

How much? If SQLite is pegged, it’s usually one core’s worth, so (for example) CPU is 25% if four cores.

CPU consumption was about 8.x %, system has 20 logical processors. Note that in task manager two items were displayed for Duplicati, having the mentioned consumption in summary:

@ts678

There’s a sixth that popped out somehow. I wonder if this last one was after your “seven hours” activities?

I think at that time I had not yet experimented with the pause/resume functionality but probably asked for an abort more than one time. When I saw the sixth put (file size: 9 KB) I hoped that this was an indicator of progress.
For completeness, I’ll mention that on the first attempt, also a single file was put at the very end after hours of no visible activity (but that may be another story, as that run had ended with the RetiredException and not due to me shutting down the system).

@ts678

Although some of this might be the general job log messiness that’s improved in next Beta, job logs are also in the database, so any unclean end can roll back the logs, along with all of the other uncommitted changes.

Good to know! I initially planned to make a copy of the -journal file before restarting Duplicati, but while writing my first post here I wanted to look up some data in the GUI and forgot that the -journal file may get lost when starting Duplicati or interacting with the backup job.

@ts678

I already asked about the trailing sixth entry, but as this is also in the database, maybe you got a clean close.

How can I find that out? Will Duplicati notify me in the next job run, if the close was not clean?

@ts678

Was the CPU idle then? One other way to get a slow GUI is to open lots of tabs to Duplicati (clogs browser).

Yes, it was no more than 0.x %. And yes to the second question, I had probably 6-8 Duplicati tabs open at that time.

@ts678

I suppose you could check the USB-connected remote. Is job 2 (which works) on same or different than 1?
Export As Command-line will get URL. Edit to empty folder for Duplicati.CommandLine.BackendTester.exe

You could use the Verify files button if you like, but it’s the same test as runs before and after a backup. Repair button can do more powerful changes (which is nice if it helps, but worse if it hurts), but maybe wait.

Thanks for the suggestion, I checked that one. I had to modify the path structure retrieved from the export option not only to point to an empty folder but also remove all backslashes (even the escaped ones) and replace them with single slashed, otherwise the command would fail with

Unittest failed: System.ArgumentException: Illegal characters in path.

Summary

PS C:\Program Files\Duplicati 2> .\Duplicati.CommandLine.BackendTester.exe “file://A:/butester/”
Starting run no 0
Generating file 0 (4.29 MB)
Generating file 1 (47.54 MB)
Generating file 2 (11.83 MB)
Generating file 3 (28.12 MB)
Generating file 4 (17.36 MB)
Generating file 5 (29.59 MB)
Generating file 6 (21.79 MB)
Generating file 7 (29.20 MB)
Generating file 8 (28.38 MB)
Generating file 9 (1.02 MB)
Uploading wrong files …
Generating file 10 (1.24 KB)
Uploading file 0, 1.24 KB … done!
Uploading file 0, 1.24 KB … done!
Uploading file 9, 1.24 KB … done!
Uploading files …
Uploading file 0, 4.29 MB … done!
Uploading file 1, 47.54 MB … done!
Uploading file 2, 11.83 MB … done!
Uploading file 3, 28.12 MB … done!
Uploading file 4, 17.36 MB … done!
Uploading file 5, 29.59 MB … done!
Uploading file 6, 21.79 MB … done!
Uploading file 7, 29.20 MB … done!
Uploading file 8, 28.38 MB … done!
Uploading file 9, 1.02 MB … done!
Renaming file 1 from BQr2ShRDtmW7VoNSynNrVcTM65BmgD1jyZzJUmB8EzeTGmcSkqPcDT4z to ewVd6hzYyOApURs3YxtZ3ab553I1rDJre4GBc7blywc2
Verifying file list …
Downloading files
Downloading file 0 … done
Checking hash … done
Downloading file 1 … done
Checking hash … done
Downloading file 2 … done
Checking hash … done
Downloading file 3 … done
Checking hash … done
Downloading file 4 … done
Checking hash … done
Downloading file 5 … done
Checking hash … done
Downloading file 6 … done
Checking hash … done
Downloading file 7 … done
Checking hash … done
Downloading file 8 … done
Checking hash … done
Downloading file 9 … done
Checking hash … done
Deleting files…
Checking retrieval of non-existent file…
*** Retrieval of non-existent file failed: System.IO.FileNotFoundException: Could not find file ‘\?\A:\butester\NonExistentFile-ab097135-39ae-400d-9983-ed6ba34e8028’.
File name: ‘\?\A:\butester\NonExistentFile-ab097135-39ae-400d-9983-ed6ba34e8028’
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at Duplicati.CommandLine.BackendTester.Program.Run(List1 args, Dictionary2 options, Boolean first)
*** Retrieval of non-existent file should have failed with FileMissingException
Checking quota…
Free Space: 75.99 GB
Total Space: 300.00 GB
Checking DNS names used by this backend…
No DNS names reported
Starting run no 1
Generating file 0 (23.82 MB)
Generating file 1 (26.22 MB)
Generating file 2 (14.66 MB)
Generating file 3 (24.22 MB)
Generating file 4 (46.10 MB)
Generating file 5 (12.46 MB)
Generating file 6 (8.35 MB)
Generating file 7 (9.44 MB)
Generating file 8 (1.10 MB)
Generating file 9 (6.58 MB)
Uploading wrong files …
Generating file 10 (1.80 KB)
Uploading file 0, 1.80 KB … done!
Uploading file 0, 1.80 KB … done!
Uploading file 9, 1.80 KB … done!
Uploading files …
Uploading file 0, 23.82 MB … done!
Uploading file 1, 26.22 MB … done!
Uploading file 2, 14.66 MB … done!
Uploading file 3, 24.22 MB … done!
Uploading file 4, 46.10 MB … done!
Uploading file 5, 12.46 MB … done!
Uploading file 6, 8.35 MB … done!
Uploading file 7, 9.44 MB … done!
Uploading file 8, 1.10 MB … done!
Uploading file 9, 6.58 MB … done!
Renaming file 1 from ae3fTyJchR71pWIA8q3MGUWFsz6Ysgt5xPnbTI6ZbdBWKOEkvQd to AolDKHKaykelH2ZzGqU7TfsWhqtAqvJxzNAFnLfNhAOYKikZ36
Verifying file list …
Downloading files
Downloading file 0 … done
Checking hash … done
Downloading file 1 … done
Checking hash … done
Downloading file 2 … done
Checking hash … done
Downloading file 3 … done
Checking hash … done
Downloading file 4 … done
Checking hash … done
Downloading file 5 … done
Checking hash … done
Downloading file 6 … done
Checking hash … done
Downloading file 7 … done
Checking hash … done
Downloading file 8 … done
Checking hash … done
Downloading file 9 … done
Checking hash … done
Deleting files…
Checking retrieval of non-existent file…
*** Retrieval of non-existent file failed: System.IO.FileNotFoundException: Could not find file ‘\?\A:\butester\NonExistentFile-3d8f9c3d-dc06-4913-9b0e-1bb8668d52a0’.
File name: ‘\?\A:\butester\NonExistentFile-3d8f9c3d-dc06-4913-9b0e-1bb8668d52a0’
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at Duplicati.CommandLine.BackendTester.Program.Run(List1 args, Dictionary2 options, Boolean first)
*** Retrieval of non-existent file should have failed with FileMissingException
Checking quota…
Free Space: 75.99 GB
Total Space: 300.00 GB
Checking DNS names used by this backend…
No DNS names reported
Starting run no 2
Generating file 0 (14.39 MB)
Generating file 1 (11.07 MB)
Generating file 2 (31.78 MB)
Generating file 3 (48.75 MB)
Generating file 4 (15.62 MB)
Generating file 5 (17.68 MB)
Generating file 6 (32.19 MB)
Generating file 7 (502.74 KB)
Generating file 8 (6.69 MB)
Generating file 9 (32.79 MB)
Uploading wrong files …
Generating file 10 (1.75 KB)
Uploading file 0, 1.75 KB … done!
Uploading file 0, 1.75 KB … done!
Uploading file 9, 1.75 KB … done!
Uploading files …
Uploading file 0, 14.39 MB … done!
Uploading file 1, 11.07 MB … done!
Uploading file 2, 31.78 MB … done!
Uploading file 3, 48.75 MB … done!
Uploading file 4, 15.62 MB … done!
Uploading file 5, 17.68 MB … done!
Uploading file 6, 32.19 MB … done!
Uploading file 7, 502.74 KB … done!
Uploading file 8, 6.69 MB … done!
Uploading file 9, 32.79 MB … done!
Renaming file 1 from 5Q7IfvafCggomPuIoyRDCDM9rgP11 to uAvFJsmu6ltrkRoeVsjEMD1JbF99WqebnTdnFsY4k7gn4
Verifying file list …
Downloading files
Downloading file 0 … done
Checking hash … done
Downloading file 1 … done
Checking hash … done
Downloading file 2 … done
Checking hash … done
Downloading file 3 … done
Checking hash … done
Downloading file 4 … done
Checking hash … done
Downloading file 5 … done
Checking hash … done
Downloading file 6 … done
Checking hash … done
Downloading file 7 … done
Checking hash … done
Downloading file 8 … done
Checking hash … done
Downloading file 9 … done
Checking hash … done
Deleting files…
Checking retrieval of non-existent file…
*** Retrieval of non-existent file failed: System.IO.FileNotFoundException: Could not find file ‘\?\A:\butester\NonExistentFile-8ba71f6f-87f8-4afd-a247-62f9535df396’.
File name: ‘\?\A:\butester\NonExistentFile-8ba71f6f-87f8-4afd-a247-62f9535df396’
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at Duplicati.CommandLine.BackendTester.Program.Run(List1 args, Dictionary2 options, Boolean first)
*** Retrieval of non-existent file should have failed with FileMissingException
Checking quota…
Free Space: 75.99 GB
Total Space: 300.00 GB
Checking DNS names used by this backend…
No DNS names reported
Starting run no 3
Generating file 0 (28.11 MB)
Generating file 1 (31.53 MB)
Generating file 2 (36.36 MB)
Generating file 3 (6.13 MB)
Generating file 4 (10.85 MB)
Generating file 5 (39.52 MB)
Generating file 6 (15.27 MB)
Generating file 7 (31.30 MB)
Generating file 8 (7.28 MB)
Generating file 9 (10.01 MB)
Uploading wrong files …
Generating file 10 (1.36 KB)
Uploading file 0, 1.36 KB … done!
Uploading file 0, 1.36 KB … done!
Uploading file 9, 1.36 KB … done!
Uploading files …
Uploading file 0, 28.11 MB … done!
Uploading file 1, 31.53 MB … done!
Uploading file 2, 36.36 MB … done!
Uploading file 3, 6.13 MB … done!
Uploading file 4, 10.85 MB … done!
Uploading file 5, 39.52 MB … done!
Uploading file 6, 15.27 MB … done!
Uploading file 7, 31.30 MB … done!
Uploading file 8, 7.28 MB … done!
Uploading file 9, 10.01 MB … done!
Renaming file 1 from b9LVIyaTjSJy4auWH2zyjav4icecPU4wMHxXNJ6k7ISJR4tZbnF5SEGb to agYUTGV5scssHT2sdALunEbVuuICGSqfDW7UhdMxjlT1VOl4rMHNPgJFeczz8tyIU
Verifying file list …
Downloading files
Downloading file 0 … done
Checking hash … done
Downloading file 1 … done
Checking hash … done
Downloading file 2 … done
Checking hash … done
Downloading file 3 … done
Checking hash … done
Downloading file 4 … done
Checking hash … done
Downloading file 5 … done
Checking hash … done
Downloading file 6 … done
Checking hash … done
Downloading file 7 … done
Checking hash … done
Downloading file 8 … done
Checking hash … done
Downloading file 9 … done
Checking hash … done
Deleting files…
Checking retrieval of non-existent file…
*** Retrieval of non-existent file failed: System.IO.FileNotFoundException: Could not find file ‘\?\A:\butester\NonExistentFile-90ce62c9-0f8a-4938-82a8-3c5fe065ab78’.
File name: ‘\?\A:\butester\NonExistentFile-90ce62c9-0f8a-4938-82a8-3c5fe065ab78’
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at Duplicati.CommandLine.BackendTester.Program.Run(List1 args, Dictionary2 options, Boolean first)
*** Retrieval of non-existent file should have failed with FileMissingException
Checking quota…
Free Space: 75.99 GB
Total Space: 300.00 GB
Checking DNS names used by this backend…
No DNS names reported
Starting run no 4
Generating file 0 (32.57 MB)
Generating file 1 (49.08 MB)
Generating file 2 (46.68 MB)
Generating file 3 (2.41 MB)
Generating file 4 (28.68 MB)
Generating file 5 (15.70 MB)
Generating file 6 (7.31 MB)
Generating file 7 (4.86 MB)
Generating file 8 (6.38 MB)
Generating file 9 (40.79 MB)
Uploading wrong files …
Generating file 10 (1.32 KB)
Uploading file 0, 1.32 KB … done!
Uploading file 0, 1.32 KB … done!
Uploading file 9, 1.32 KB … done!
Uploading files …
Uploading file 0, 32.57 MB … done!
Uploading file 1, 49.08 MB … done!
Uploading file 2, 46.68 MB … done!
Uploading file 3, 2.41 MB … done!
Uploading file 4, 28.68 MB … done!
Uploading file 5, 15.70 MB … done!
Uploading file 6, 7.31 MB … done!
Uploading file 7, 4.86 MB … done!
Uploading file 8, 6.38 MB … done!
Uploading file 9, 40.79 MB … done!
Renaming file 1 from k9YiClFB47tANJjahj9Lan6TvISV7Rgn3JQLAhNyhNo to TPYKYQCbRRYbkRxMf5lkXpN3GVdH8DfqJkDczjhG2fxFovy1XDOQZFeEb6ddM5dnMIBaoun5
Verifying file list …
Downloading files
Downloading file 0 … done
Checking hash … done
Downloading file 1 … done
Checking hash … done
Downloading file 2 … done
Checking hash … done
Downloading file 3 … done
Checking hash … done
Downloading file 4 … done
Checking hash … done
Downloading file 5 … done
Checking hash … done
Downloading file 6 … done
Checking hash … done
Downloading file 7 … done
Checking hash … done
Downloading file 8 … done
Checking hash … done
Downloading file 9 … done
Checking hash … done
Deleting files…
Checking retrieval of non-existent file…
*** Retrieval of non-existent file failed: System.IO.FileNotFoundException: Could not find file ‘\?\A:\butester\NonExistentFile-aee56595-3296-4545-a543-a9bff21a09da’.
File name: ‘\?\A:\butester\NonExistentFile-aee56595-3296-4545-a543-a9bff21a09da’
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.File.InternalCopy(String sourceFileName, String destFileName, Boolean overwrite, Boolean checkHost)
at Duplicati.CommandLine.BackendTester.Program.Run(List1 args, Dictionary2 options, Boolean first)
*** Retrieval of non-existent file should have failed with FileMissingException
Checking quota…
Free Space: 75.99 GB
Total Space: 300.00 GB
Checking DNS names used by this backend…
No DNS names reported
Unittest complete!

Besides the expected FileMissingException which is in fact a FileNotFoundException (probably not really an issue?) I do not see any problems with the remote. Do you agree?

@ts678

You might want to set a log-file= log-file-log-level=verbose, or profiling if you can stand a very big log when you try backup next. For a minimal log, information or retry is good. Maybe it’ll just run OK next time…

It’d be a good idea to inspect your retention on Options screen to see if you want to halt deletes temporarily. Turning no-auto-compact for now would make extra sure that the not-sure-it’s-healthy remote gets less use.

I have added these options with log level verbose, compact disabled and all data now being retained.
Having done that, I will give the job one more try, hopefully the changed options will result in some clue for further investigation.

Thanks for your help so far!