How to recover from aborted/stuck/crashed backup run?

ts678 · April 27, 2024, 12:00am

I searched for this phrase with Google, and found a couple of references. This most interesting one was:

github.com/duplicati/duplicati

Catch RetiredException in BackupHandler to avoid StopNow race condition

duplicati:master ← Jojo-1000:fix-backend-requests-retired

opened 06:01PM - 22 Feb 24 UTC

Jojo-1000

+14 -10

Closes #4980 There seems to be no way to reliably check whether the channel …is retired, so I handle the `RetiredException` which should fix the random test failures for `StopNow` that occur sometimes (most recently [in this build](https://ci.appveyor.com/project/kenkendk/duplicati/builds/49202956#L5932)). The [CoCoL examples](https://libraries.io/nuget/CoCoL) also uses the exception to handle retired channels.

github.com/duplicati/duplicati

Race condition when checking if retired (StopNow)

opened 11:59AM - 26 Jun 23 UTC

closed 12:33PM - 04 Mar 24 UTC

Jojo-1000

- [x] I have searched open and closed issues for duplicates. - [x] I have searc…hed the [forum](https://forum.duplicati.com) for related topics. ---------------------------------------- ## Environment info - **Duplicati version**: master (d0f1498bd41b151d8512fd2acb57739f6a05587f) - **Operating system**: Replace me - **Backend**: Replace me ## Description There is a race condition when checking if the upload channel is expired which was found by the `StopNow` unit test. ## Steps to reproduce The error appeared in [this run](https://github.com/duplicati/duplicati/actions/runs/5359093052/jobs/9745427625?pr=4978) of the StopNow unit test. Only the Windows test failed, not the other OS. It is not reproducible, because it depends on the exact thread execution order. However, I was able to emulate the error with a slightly different stack trace: - Set breakpoint in Backup handler on line 383 - Run StopNow unit test - Skip the if statement by right click, Set next statement to line 387 - Continue debugging https://github.com/duplicati/duplicati/blob/d0f1498bd41b151d8512fd2acb57739f6a05587f/Duplicati/Library/Main/Operation/BackupHandler.cs#L381-L389 The cause is that another thread can call `StopNow()` on the operation between the check and the `WriteAsync` call. - **Actual result**: `uploadtarget.WriteAsync()` throws an exception: `CoCoL.RetiredException : The channel "BackendRequests" is retired` - **Expected result**: The check should be atomic (or at least wrapped in a try catch) and the function should return -1. ## Debug log Log of failed unit test: ``` 1) Error : Duplicati.UnitTest.DisruptionTests.StopNow CoCoL.RetiredException : The channel "BackendRequests" is retired at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at CoCoL.Channel`1.<WriteAsync>d__32.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<FlushBackend>d__19.MoveNext() in D:\a\duplicati\duplicati\Duplicati\Library\Main\Operation\BackupHandler.cs:line 390 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__20.MoveNext() in D:\a\duplicati\duplicati\Duplicati\Library\Main\Operation\BackupHandler.cs:line 589 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task) at Duplicati.Library.Main.Controller.<>c__DisplayClass14_0.<Backup>b__0(BackupResults result) in D:\a\duplicati\duplicati\Duplicati\Library\Main\Controller.cs:line 126 at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method) in D:\a\duplicati\duplicati\Duplicati\Library\Main\Controller.cs:line 499 at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter) in D:\a\duplicati\duplicati\Duplicati\Library\Main\Controller.cs:line 124 at System.Threading.Tasks.Task`1.InnerInvoke() at System.Threading.Tasks.Task.Execute() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Duplicati.UnitTest.DisruptionTests.<StopNow>d__10.MoveNext() in D:\a\duplicati\duplicati\Duplicati\UnitTest\DisruptionTests.cs:line 611 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at NUnit.Framework.Internal.TaskAwaitAdapter.GenericAdapter`1.BlockUntilCompleted() in D:\a\1\s\src\NUnitFramework\framework\Internal\TaskAwaitAdapter.cs:line 95 at NUnit.Framework.Internal.AsyncToSyncAdapter.Await(Func`1 invoke) in D:\a\1\s\src\NUnitFramework\framework\Internal\AsyncToSyncAdapter.cs:line 60 at NUnit.Framework.Internal.Commands.TestMethodCommand.Execute(TestExecutionContext context) in D:\a\1\s\src\NUnitFramework\framework\Internal\Commands\TestMethodCommand.cs:line 64 at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.<>c__DisplayClass1_0.<Execute>b__0() in D:\a\1\s\src\NUnitFramework\framework\Internal\Commands\BeforeAndAfterTestCommand.cs:line 58 at NUnit.Framework.Internal.Commands.BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) in D:\a\1\s\src\NUnitFramework\framework\Internal\Commands\BeforeAndAfterTestCommand.cs:line 73 ``` Log of reproduction steps: ``` Message: CoCoL.RetiredException : The channel "BackendRequests" is retired Stack Trace: TaskAwaiter.ThrowForNonSuccess(Task task) TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) TaskAwaiter.ValidateEnd(Task task) <WriteAsync>d__32.MoveNext() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- TaskAwaiter.ThrowForNonSuccess(Task task) TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) ConfiguredTaskAwaiter.GetResult() <FlushBackend>d__19.MoveNext() line 389 --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- TaskAwaiter.ThrowForNonSuccess(Task task) TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) ConfiguredTaskAwaiter.GetResult() <RunAsync>d__20.MoveNext() line 590 --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- ChannelExtensions.WaitForTaskOrThrow(Task task) BackupHandler.Run(String[] sources, IFilter filter, CancellationToken token) line 349 <>c__DisplayClass14_0.<Backup>b__0(BackupResults result) line 128 Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method) line 499 Controller.Backup(String[] inputsources, IFilter filter) line 124 <>c__DisplayClass10_0.<StopNow>b__0() line 605 Task`1.InnerInvoke() Task.Execute() --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- TaskAwaiter.ThrowForNonSuccess(Task task) TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) ConfiguredTaskAwaiter.GetResult() <StopNow>d__10.MoveNext() line 612 --- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde --- TaskAwaiter.ThrowForNonSuccess(Task task) TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) GenericAdapter`1.BlockUntilCompleted() line 95 NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaitable) line 64 AsyncToSyncAdapter.Await(Func`1 invoke) line 54 TestMethodCommand.RunTestMethod(TestExecutionContext context) line 81 TestMethodCommand.Execute(TestExecutionContext context) line 64 <>c__DisplayClass1_0.<Execute>b__0() line 58 BeforeAndAfterTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action) line 73 ```

2.0.7.101_canary_2024-03-08

Fixed exception happening when stopping the running backup, thanks @Jojo-1000

which matches:

So this is possibly also fixed in next Beta.