Backup stalled and now will not run


#1

After 4 days backup stalled to Google Drive. Tried abort after file finished, but of course it was not gonna finish. So I told it to abort now.

Then told it to run the backup again, came back with a database locked error

Restarted Duplicati and am now getting the following error:

Failed: One or more errors occurred.
Details: System.AggregateException: One or more errors occurred. ---> System.AggregateException: Detected non-empty blocksets with no associated blocks! ---> System.Exception: Detected non-empty blocksets with no associated blocks!
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x0017e] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x0102c] in <0ce58d578b8642d49036dc15fbad38f1>:0 
   --- End of inner exception stack trace ---
  at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x0005d] in <6973ce2780de4b28aaa2c5ffc59993b1>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x00008] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass13_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00035] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011d] in <0ce58d578b8642d49036dc15fbad38f1>:0 
---> (Inner Exception #0) System.AggregateException: Detected non-empty blocksets with no associated blocks! ---> System.Exception: Detected non-empty blocksets with no associated blocks!
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x0017e] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0 
   --- End of inner exception stack trace ---
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x0102c] in <0ce58d578b8642d49036dc15fbad38f1>:0 
---> (Inner Exception #0) System.Exception: Detected non-empty blocksets with no associated blocks!
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x0017e] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0 <---

---> (Inner Exception #1) System.AggregateException: One or more errors occurred. ---> System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
  at Duplicati.Library.Logging.Log.get_CurrentScope () [0x0004d] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x0001e] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Logging.Log.WriteWarningMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c.<Run>b__1_3 (System.String rootpath, System.String errorpath, System.Exception ex) [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Utility.Utility+<EnumerateFileSystemEntries>d__23.MoveNext () [0x001d9] in <5336a8b903594fa5ae5a9692dae4b7fe>:0 
  at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x0006f] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<ExpandWorkList>d__4.MoveNext () [0x000da] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c__DisplayClass1_0+<<Run>b__0>d.MoveNext () [0x00263] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at CoCoL.AutomationExtensions+<RunTask>d__10`1[T].MoveNext () [0x000cc] in <6973ce2780de4b28aaa2c5ffc59993b1>:0 
   --- End of inner exception stack trace ---
---> (Inner Exception #0) System.Exception: Unable to find log in lookup table, this may be caused by attempting to transport call contexts between AppDomains (eg. with remoting calls)
  at Duplicati.Library.Logging.Log.get_CurrentScope () [0x0004d] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Logging.Log.WriteMessage (Duplicati.Library.Logging.LogMessageType type, System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x0001e] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Logging.Log.WriteWarningMessage (System.String tag, System.String id, System.Exception ex, System.String message, System.Object[] arguments) [0x00000] in <a0d17c5f1b8942efac55f621de8cc00b>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c.<Run>b__1_3 (System.String rootpath, System.String errorpath, System.Exception ex) [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Utility.Utility+<EnumerateFileSystemEntries>d__23.MoveNext () [0x001d9] in <5336a8b903594fa5ae5a9692dae4b7fe>:0 
  at System.Linq.Enumerable+SelectManySingleSelectorIterator`2[TSource,TResult].MoveNext () [0x0006f] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<ExpandWorkList>d__4.MoveNext () [0x000da] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess+<>c__DisplayClass1_0+<<Run>b__0>d.MoveNext () [0x00263] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at CoCoL.AutomationExtensions+<RunTask>d__10`1[T].MoveNext () [0x000cc] in <6973ce2780de4b28aaa2c5ffc59993b1>:0 <---
<---
<---

Log data:

2018-07-04 15:16:09 -06 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Exception: Detected non-empty blocksets with no associated blocks!
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (System.Int64 blocksize, System.Int64 hashsize, System.Boolean verifyfilelists, System.Data.IDbTransaction transaction) [0x0017e] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () [0x00000] in <0ce58d578b8642d49036dc15fbad38f1>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () [0x00038] in <0ce58d578b8642d49036dc15fbad38f1>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0003e] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x00028] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x00008] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <71d8ad678db34313b7f718a414dfcb25>:0 
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () [0x003c9] in <0ce58d578b8642d49036dc15fbad38f1>:0 

I have repair and compacted, they come back with no problems, but run backup comes back with the above. Am not sure what to look for

Thankls
Myk


#2

this is what the web log says when it errors out, up above was email

MainOperation: Test
Verifications: []
ParsedResult: Success
Version: 2.0.3.9 (2.0.3.9_canary_2018-06-30)
EndTime: 7/4/2018 3:12:07 PM (1530738727)
BeginTime: 7/4/2018 3:11:59 PM (1530738719)
Duration: 00:00:08.7049700
Messages: [
    2018-07-04 15:11:59 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started,
    2018-07-04 15:11:59 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
    2018-07-04 15:12:07 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (736 bytes)
]
Warnings: []
Errors: []
TaskReader:
    ProgressAsync:
        Result: True
        Factory:
            CancellationToken:
                IsCancellationRequested: False
                CanBeCanceled: False
                WaitHandle:
                    Handle: 10746008
                    SafeWaitHandle:
                        IsInvalid: False
                        IsClosed: False
            Scheduler: null
            CreationOptions: None
            ContinuationOptions: None
        Id: 5
        Exception: null
        Status: RanToCompletion
        IsCanceled: False
        IsCompleted: True
        IsCompletedSuccessfully: True
        CreationOptions: None
        AsyncState: null
        IsFaulted: False
    TransferProgressAsync:
        Result: True
        Id: 6
        Exception: null
        Status: RanToCompletion
        IsCanceled: False
        IsCompleted: True
        IsCompletedSuccessfully: True
        CreationOptions: None
        AsyncState: null
        IsFaulted: False
BackendStatistics:
    RemoteCalls: 1
    BytesUploaded: 0
    BytesDownloaded: 0
    FilesUploaded: 0
    FilesDownloaded: 0
    FilesDeleted: 0
    FoldersCreated: 0
    RetryAttempts: 0
    UnknownFileSize: 0
    UnknownFileCount: 0
    KnownFileCount: 736
    KnownFileSize: 96512708160
    LastBackupDate: 1/1/0001 12:00:00 AM (-62135571600)
    BackupListCount: 0
    TotalQuotaSpace: 12675366215949
    FreeQuotaSpace: 1680249938189
    AssignedQuotaSpace: -1
    ReportedQuotaError: False
    ReportedQuotaWarning: False
    ParsedResult: Success
    Version: 2.0.3.9 (2.0.3.9_canary_2018-06-30)
    Messages: [
        2018-07-04 15:11:59 -06 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started,
        2018-07-04 15:11:59 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
        2018-07-04 15:12:07 -06 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (736 bytes)
    ]
    Warnings: []
    Errors: []
    TaskReader: null

#3

This sounds like the same thing reported here:

Unfortunately, I don’t know that a specific cause for this has been isolated. @Pectojin, do I recall correctly that you figured out the underlying issue (but not the cause)?

(By the way, I edited your posts by adding “~~~” before and after the log and error text to make them more readable.)


#4

Hmm, I don’t recall stumbling upon this :confused:

It appears to be from this part of the code duplicati/LocalDatabase.cs at 6691d67b824287777f271d48a1e66fb3f4ec40aa · duplicati/duplicati · GitHub

It’s just a SQL query check validating the database, so it should be possible to resolve by recreating the database if it’s just a DB mistake.

I’m not sure how the DB would end in this state, though. If the backend was corrupted I would expect an other error…


#5

I try to delete and recreate the database for that backup and get

No filelists found on the remote destination

and if I go look at the Google Drive folder, there are the hundreds of files from the 4 days it worked on this backup.

Only way I have been able to in the past (3trd time this has happened) is delete the config/location and start over

When it initially stalled and had to stop it now, it errored out with database locked. I waited several hours, still locked and finally restarted the docker and now have the above error again


#6

That’s what I recall reading elsewhere - the issue doesn’t go away (or does for only a short while) even with a database recreate, so there’s definitely something other than a gremlin that’s causing it.

On the flip side, very few people have reported the issue so whatever does cause it isn’t common…


#7

Few people yes, but this is the 3rd time its happen to me and have had to throw out the backup. My upload speed here at home is not the greatest, so I have yet to get a complete backup for this configuration :frowning:


#8

Just got another database locked error and am gonna loose the backup after 1 days worth of upload :frowning:

What do I need to look for to help try and track this down, I cant get any decent sized backups to complete.


#9

@kenkendk, I’m pretty stumped on this one. Do you have any ideas?