Fatal error System.Exception: Detected non-empty blocksets with no associated blocks!


#1

I created a new backup from one hard drive to a hard drive in my little network. It started and run appr. an hour without issues. Then I rebooted my computer. But now I can’t start this backup any more. Errors see below.

Windows 7 64 bit, Duplicati - 2.0.3.7_canary_2018-06-17

2018-06-27 18:07:33 +02 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started
2018-06-27 18:07:33 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.Exception: Detected non-empty blocksets with no associated blocks!
   bei Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency(Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction)
   bei Duplicati.Library.Main.Operation.Backup.BackupDatabase.<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0()
   bei Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass6_0.<RunOnMain>b__0()
   bei Duplicati.Library.Main.Operation.Common.SingleRunner.<>c__DisplayClass5_0`1.<<DoRunOnMain>b__1>d.MoveNext()
--- Ende der Stapelüberwachung vom vorhergehenden Ort, an dem die Ausnahme ausgelöst wurde ---
   bei System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   bei System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

Repair of the database didn’t help:
[Information-Duplicati.Library.Main.Operation.RepairHandler-DatabaseIsSynchronized]: Destination and database are synchronized, not making any changes


V2.0.3.8 Linux - Detected non-empty blocksets with no associated blocks!
Fatal error: Detected non-empty blocksets with no associated blocks
Backup stalled and now will not run
#2

Second attempt:

I deleted the database and started Duplicati again. It ended with following error:

2018-06-27 23:59:54 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-ifbb1ef804a184fa396e7dcbd16782ffe.dindex.zip
2018-06-27 23:59:54 +02 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraUnknownFile]: Extra unknown file: duplicati-iffea6c4773fa4d7685501f3b61cfb508.dindex.zip
2018-06-27 23:59:54 +02 - [Error-Duplicati.Library.Main.Operation.FilelistProcessor-ExtraRemoteFiles]: Found 418 remote files that are not recorded in local storage, please run repair
2018-06-27 23:59:54 +02 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
Duplicati.Library.Interface.UserInformationException: Found 418 remote files that are not recorded in local storage, please run repair
   bei Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
   bei Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify(BackendManager backend, String protectedfile)
   bei Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

V2.0.3.8 Linux - Detected non-empty blocksets with no associated blocks!
#3

3rd attempt (No filelists found on the remote destination):

Started to recreate the database. It ended with following error:

2018-06-28 00:06:56 +02 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started
2018-06-28 00:06:57 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()
2018-06-28 00:06:57 +02 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (418 bytes)

=======> ERROR from GUI: Error while running PC1--Name--Desktop auf Q: No filelists found on the remote destination

Do you recommend as a first step to downgrade to Beta 2.0.3.3 instead of using Canary 2.0.3.7??


#5

Everything worked, but immediately after update (to 2.0.3.8) this error message is given and backup won’t run.

So something is different, and seems to be broken too.

Software as usual, I love it when people ask if this new update will fix issues. Sure it will fix some, and you’ll get many interesting new ones too. - But that’s the way it is.

Fatal error => System.Exception: Detected non-empty blocksets with no associated blocks!
  at Duplicati.Library.Main.Database.LocalDatabase.VerifyConsistency (Int64 blocksize, Int64 hashsize, Boolean verifyfilelists, IDbTransaction transaction) <0x4109d4c0 + 0x006c7> in <filename unknown>:0 
  at Duplicati.Library.Main.Operation.Backup.BackupDatabase+<>c__DisplayClass31_0.<VerifyConsistencyAsync>b__0 () <0x4109b360 + 0x0003b> in <filename unknown>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass6_0.<RunOnMain>b__0 () <0x4109b320 + 0x00019> in <filename unknown>:0 
  at Duplicati.Library.Main.Operation.Common.SingleRunner+<>c__DisplayClass5_0`1+<<DoRunOnMain>b__1>d[T].MoveNext () <0x4109ae60 + 0x000e7> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7fa54f8016d0 + 0x00029> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7fa54f7ff6b0 + 0x000a7> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7fa54f7ff630 + 0x0006b> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7fa54f7ff5e0 + 0x0003a> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x7fa54f7ff5c0 + 0x00012> in <filename unknown>:0 
  at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__19.MoveNext () <0x4103d520 + 0x011de> in <filename unknown>:100:

Linux is mentioned in the subject, because I did run backup on several Windows machines, and I didn’t get this error message. Let’s try with another Linux computer too… And see if the same happens.

Interesting, no problem with alternate system. Anyway, what’s the root cause and how it should be fixed?

Sounds like some older version could have left something in “unclean” state in the database, and this new version doesn’t like that.

It seems that some users experienced this already with the previous version:


#6

@DUser & @Sami_Lehtinen - I’ve merged your topics as they are both the same thing. (@Sami_Lehtinen, I tweaked your post to make it clear you’re on 2.0.3.8.)

I suspect this won’t improve things, but what happens if you add --npo-backend-verification=true?

--no-backend-verification
If this flag is set, the local database is not compared to the remote filelist on startup. The intended usage for this option is to work correctly in cases where the filelisting is broken or unavailable.
Default value: “false”


Unfortunately, I’m having trouble finding what I thought was an even earlier instance of this for which @Pectojin had a solution (I think).

Until he “gets here” all I can say is that from what I recall there’s an issue where a destination compacting that happens resulting in a file being deleted (as it should) but one part of the local database doesn’t get updated to indicate the delete happened.

Normally, a database recreate would resolve this but part of the issue is that since the database record isn’t deleted, neither is a destination file (either dlist or dindex, I forget which). The results of this are that even if you recreate the database, it comes back still broken because the orphaned destination file still reports that there should be an empty dlist file, which then re-screws up the database.


#7

Thank you very much for your reply. Unfortunately I can’t help any more because I switched to Duplicacy.


#8

Updating to version v2.0.3.9 didn’t change a thing, backups still halt with fatal error.


#9

I now get this error on two of my backup sets. The only fix seems to be to delete the whole backup and re-run it, which is not viable at all! Hope we can pinpoint and fix this one soon!


#10

I’m now getting this same error on my 2.0.4.2_experimental_2018-11-12 system (Windows 10 laptop). I tried a Verify (got this error again) and Database Repair. Is there a fix?


#11

I now get the same error. Repairing the database does not solve the issue. Is there a command to launch to get what orphaned files are causing the issue on the remote destination? Would removing them solve the issue?


#12

I was able to get again functionality since I also backup separately the local job database. So I restored it to the date at which the main job was last functioning, and remotely deleted all dblock and dlist files following said date. I then repaired the database and jobs are now working again.

Hopefully I neither needed a database recreate, nor to start the job from scratch. I speculate the issue was due to crappy internet connection, and inability by Duplicati to handle this.

However, given this kind of issues in Duplicati, I find it crucial to have a separate backup of the local job database, and to disable the automatic compact feature (using --no-auto-compact). Can anybody confirm that, this way, remote dblock and dlist files are never modified again, unless manually requested? (Besides files affected by retention policy, of course)

(This is on beta 2.0.4.5)


Fatal error: Detected non-empty blocksets with no associated blocks
#13

yes @gianlucabertaina there are some steps in this post on the same error: Fatal error: Detected non-empty blocksets with no associated blocks. I and another user have followed them and got similar results. No precise filenames but filenames “nearby”. Doesn’t lead to a fix, tho, just more clarity around the circumstances of what is looking like a bug in Duplicati.

It would be great to have a third set of results from these steps, if you have the time.


#14

Hi, I would be happy to contribute, but I was not so systematic in my
debugging, so I already deleted the faulty database, once the restored
one was proven to work.


#15

I have this issue as well on one of my backups. It just started recently, but I didn’t notice the update to 2.0.4.5 beta causing the issue. The profile worked for a while after I updated I think. First, I noticed some errors that my file name was too long and backups were failing. Then, I noticed it was having issues backing up my new outlook OST file. So I made a filter to exclude *.OST file extensions and re-ran the backup to get a better log of the long named file. I no longer got that error, but now got this one about non-empty blocksets.

If I can help with any testing, please let me know. I have a second backup going off-site that’s still working, so I’m not in a critical state where I can’t pause for troubleshooting.