"Unexpected number of remote volumes marked as deleted" error


#1

I am getting an “Unexpected number of remote volumes marked as deleted.” error every time I run my backup job.

Googling found several topics with similar errors, but most seemed very old or were fixed by upgrading Duplicati. Since the version upgraded to was older than what I’m running, I don’t think that would help. :slight_smile:

I have tried running the repair operation multiple times, it does not fix anything.

I have also deleted and recreated the database via the webui multiple times. Again, it did not help.

How can I fix this?

OS: Ubuntu 16.04
Duplicati: 2.0.2.1_beta_2017-08-01

Here is the sanatized output I get when running the backup command from the command line.

Reach# mono "/usr/lib/duplicati/Duplicati.CommandLine.exe" backup "ftp://goflex.lionsarch.tyria/GoFlex Home Personal/Duplicati/Reach/GoFlex Reach Home?auth-username=<ftpusername>&auth-password=password" "/home/<localusername>/" "/etc/hosts" "/var/lib/mysql/" "/opt/databasedumps/" --backup-name="GoFlex Reach Home" --dbpath="/root/.config/Duplicati/AHGLUNAFNK.sqlite" --encryption-module="aes" --compression-module="zip" --dblock-size="50mb" --keep-time="3Y" --passphrase="backuppassword" --send-mail-from="no-reply-reach" --send-mail-level="All" --send-mail-password="mailpass" --send-mail-to="mailto" --send-mail-url="mailurl" --send-mail-username="mailuser" --zip-compression-level="9" --run-script-timeout="3h" --exclude-files-attributes="temporary,system" --disable-module="console-password-input" --exclude="<a bunch of exclude flags I've removed from this post."
Backup started at 12/27/2017 1:07:33 PM
Checking remote backup ...
  Listing remote folder ...
promoting uploaded complete file from Uploading to Uploaded: duplicati-20171227T191233Z.dlist.zip.aes
Expected there to be a temporary fileset for synthetic filelist (21, duplicati-b001c8ef433d745b78d76bc03be5cf0fa.dblock.zip.aes), but none was found?
Scanning local files ...
  54040 files need to be examined (10.04 GB) (still counting)
  67680 files need to be examined (10.13 GB) (still counting)
  83257 files need to be examined (10.56 GB) (still counting)
  98338 files need to be examined (7.21 GB) (still counting)
  113448 files need to be examined (7.29 GB) (still counting)
  129146 files need to be examined (9.99 GB) (still counting)
  142060 files need to be examined (10.43 GB) (still counting)
  157211 files need to be examined (10.23 GB) (still counting)
  171613 files need to be examined (7.40 GB) (still counting)
  186575 files need to be examined (5.50 GB) (still counting)
  203106 files need to be examined (6.78 GB) (still counting)
  221252 files need to be examined (6.92 GB) (still counting)
  235631 files need to be examined (7.21 GB) (still counting)
  247748 files need to be examined (7.65 GB) (still counting)
  262362 files need to be examined (8.41 GB) (still counting)
  277052 files need to be examined (8.35 GB) (still counting)
  294158 files need to be examined (8.81 GB) (still counting)
  312092 files need to be examined (9.75 GB) (still counting)
  327488 files need to be examined (14.19 GB) (still counting)
  343615 files need to be examined (18.96 GB) (still counting)
  364082 files need to be examined (87.38 GB) (still counting)
  380246 files need to be examined (87.86 GB) (still counting)
  398018 files need to be examined (88.45 GB) (still counting)
  414861 files need to be examined (88.61 GB) (still counting)
  427148 files need to be examined (89.27 GB) (still counting)
  441818 files need to be examined (114.69 GB) (still counting)
  452367 files need to be examined (118.26 GB) (still counting)
  464372 files need to be examined (216.78 GB) (still counting)
  474967 files need to be examined (222.76 GB) (still counting)
  489727 files need to be examined (224.32 GB) (still counting)
  506209 files need to be examined (230.69 GB) (still counting)
  507212 files need to be examined (233.88 GB)
  497502 files need to be examined (233.80 GB)
  487694 files need to be examined (233.75 GB)
  478752 files need to be examined (233.42 GB)
  467801 files need to be examined (233.36 GB)
  457881 files need to be examined (233.32 GB)
  449432 files need to be examined (233.29 GB)
  438388 files need to be examined (231.96 GB)
  428816 files need to be examined (231.92 GB)
  419558 files need to be examined (231.89 GB)
  407020 files need to be examined (231.87 GB)
  396889 files need to be examined (231.57 GB)
  387065 files need to be examined (231.51 GB)
  378708 files need to be examined (231.34 GB)
  371759 files need to be examined (231.00 GB)
  365839 files need to be examined (230.98 GB)
  359989 files need to be examined (230.83 GB)
  352953 files need to be examined (230.50 GB)
  345417 files need to be examined (230.06 GB)
  338649 files need to be examined (229.98 GB)
  331673 files need to be examined (229.95 GB)
  325954 files need to be examined (229.90 GB)
  319188 files need to be examined (229.81 GB)
  312426 files need to be examined (229.76 GB)
  307315 files need to be examined (229.43 GB)
  300634 files need to be examined (229.34 GB)
  292639 files need to be examined (229.09 GB)
  286077 files need to be examined (228.53 GB)
  278045 files need to be examined (228.38 GB)
  270419 files need to be examined (227.81 GB)
  261494 files need to be examined (223.95 GB)
  252545 files need to be examined (223.77 GB)
  243601 files need to be examined (222.77 GB)
  234324 files need to be examined (220.49 GB)
  226274 files need to be examined (217.91 GB)
  216232 files need to be examined (150.40 GB)
  206788 files need to be examined (150.32 GB)
  197373 files need to be examined (149.94 GB)
  187970 files need to be examined (149.72 GB)
  177859 files need to be examined (149.64 GB)
  167406 files need to be examined (149.52 GB)
  157456 files need to be examined (149.04 GB)
  147055 files need to be examined (148.74 GB)
  137373 files need to be examined (148.69 GB)
  129160 files need to be examined (148.65 GB)
  120268 files need to be examined (148.54 GB)
  112224 files need to be examined (146.92 GB)
  105536 files need to be examined (124.50 GB)
  97179 files need to be examined (122.69 GB)
  93589 files need to be examined (116.22 GB)
  81442 files need to be examined (115.70 GB)
  74303 files need to be examined (109.93 GB)
  65793 files need to be examined (17.44 GB)
  56302 files need to be examined (12.43 GB)
  55682 files need to be examined (12.39 GB)
  52433 files need to be examined (11.83 GB)
  45164 files need to be examined (11.18 GB)
  37147 files need to be examined (10.96 GB)
  29288 files need to be examined (9.42 GB)
  27422 files need to be examined (9.37 GB)
  25429 files need to be examined (9.34 GB)
  19577 files need to be examined (5.86 GB)
  10352 files need to be examined (3.63 GB)
  0 files need to be examined (0 bytes)
  Uploading file (25.27 MB) ...
  Uploading file (255.03 KB) ...
  Uploading file (37.94 MB) ...
Fatal error => Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes

System.Exception: Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes
  at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x00252] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0005e] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact) [0x0015c] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x00092] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x00860] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass16_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0030f] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0014b] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00068] in <118ad25945a24a3991f7b65e7a45ea1e>:0 
  at Duplicati.CommandLine.Commands.Backup (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x000ee] in <08be045d41054657bb406455b4cf1a83>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32]:invoke_TResult_T1_T2_T3_T4_T5 (System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x003fb] in <08be045d41054657bb406455b4cf1a83>:0 

Here is what I get when I run the repair operation from the command line:

Reach# mono "/usr/lib/duplicati/Duplicati.CommandLine.exe" repair "ftp://goflex.lionsarch.tyria/GoFlex Home Personal/Duplicati/Reach/GoFlex Reach Home?auth-username=ftpuser&auth-password=ftppass" --passphrase="password" --dbpath="/root/.config/Duplicati/AHGLUNAFNK.sqlite"
  Listing remote folder ...
Destination and database are synchronized, not making any changes

I’d post the output of the delete and recreate operation, but I’m not 100% sure how to run that from the command line. Do I just rm the database file and then run a repair?


Problem with backup after canceling it
#2

As you probably noted, database recreation can take quite a while. If you have the time available then it’s certainly worth a try, but I’d rename or move the existing database rather than delete it (just in case).

As for the actual “Unexpected number of remote volumes marked as deleted” error you’re seeing, I haven’t dealt with it before so I’m not exactly sure what’s going on. I believe it’s saying is that the number of files deleted from the local database (the filesets) didn’t mach the number of destination files that were flagged to be eventually be deleted (the volumes) during the next cleanup / compacting phase.

So in your case, there were 0 files deleted from the database, yet 1 destination file was found to be flagged for later deletion.

This is odd because code wise this should only happen if Duplicati is interrupted exactly during this block of code. Do you recall a power outage or system crash that might have happened just before you started getting this?

As for resolving it, @kees-z or @kenkendk might have a suggestion.


#3

Well, I finally had time to get back to this issue.

I went ahead and upgraded to v2.0.2.15-2.0.2.15_experimental_2018-01-03 downloaded from GitHub.

I moved the old database and ran the repair operation from the command line. It completed successfully, except for these messages partway through:

Operation Get with file duplicati-20170611T183256Z.dlist.zip.aes attempt 1 of 5 failed with message: File length is invalid => File length is invalid
  Downloading file (40.42 MB) ...
Operation Get with file duplicati-20170611T183256Z.dlist.zip.aes attempt 2 of 5 failed with message: File length is invalid => File length is invalid
  Downloading file (40.42 MB) ...
Operation Get with file duplicati-20170611T183256Z.dlist.zip.aes attempt 3 of 5 failed with message: File length is invalid => File length is invalid
  Downloading file (40.42 MB) ...
Operation Get with file duplicati-20170611T183256Z.dlist.zip.aes attempt 4 of 5 failed with message: File length is invalid => File length is invalid
  Downloading file (40.42 MB) ...
Operation Get with file duplicati-20170611T183256Z.dlist.zip.aes attempt 5 of 5 failed with message: File length is invalid => File length is invalid
  Downloading file (41.32 MB) ...
Failed to process file: duplicati-20170611T183256Z.dlist.zip.aes => File length is invalid

It ended with this:

Processing required 1 blocklist volumes
  Downloading file (49.99 MB) ...
Recreate completed, verifying the database consistency
Recreate completed, and consistency checks completed, marking database as complete

Then I ran a backup, and received the same error as before.

Here is the error message again. Just in case there’s something different I didn’t spot.

Fatal error => Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes

System.Exception: Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes
  at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x00252] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0005e] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <521f3fe476f34ea88748e97b3b984d05>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00169] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x008a5] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass17_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00036] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0014b] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00068] in <ab07a85a74244406aaefc2202f7e530f>:0 
  at Duplicati.CommandLine.Commands.Backup (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x000ee] in <4d8cdd60210946ec8cc8a22c1148fc2f>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32]:invoke_TResult_T1_T2_T3_T4_T5 (System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x003fb] in <4d8cdd60210946ec8cc8a22c1148fc2f>:0 

If you need the rest of the output for some reason, I have saved it.

It’s been months since this started happening, so I don’t remember. I actually have the exact same data being backed up to Google Drive in another job. So it hasn’t been on the top of my priority list. (The data is in a work account, so I want a local backup. Plus, I have yet to figure out why my internet connection starts losing packets whenever I try to backup to Google drive here at home. It works fine from work…)


Problem with backup after canceling it
#4

Thanks for following up on how things went for you after the version change.

At this point I think we’ll need input from somebody more familiar with the database then I am (such as @kenkendk) for how to resolve the “Unexpected number of remote volumes marked as deleted” error. :frowning:


#5

Email notified me of this post so I tried the --auto-cleanup flag. It did not help.

I’m on Duplicati - 2.0.3.3_beta_2018-04-02 and Ubuntu 18.04 now.

The end of the output:

0 files need to be examined (0 bytes)
  Uploading file (17.74 MB) ...
  Uploading file (218.09 KB) ...
  Uploading file (81.68 MB) ...
Fatal error => Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes

System.Exception: Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes
  at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x00252] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0005e] in <4bd4dbd20d094f3f8734f677798e9741>:0 
  at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <4bd4dbd20d094f3f8734f677798e9741>:0 
  at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <4bd4dbd20d094f3f8734f677798e9741>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00169] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x008e0] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller+<>c__DisplayClass17_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x00036] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0014b] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.Backup (System.String[] inputsources, Duplicati.Library.Utility.IFilter filter) [0x00068] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.CommandLine.Commands.Backup (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x000ee] in <71ed3aa6fd0e4ec4934b502e9e6ebc99>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x003fb] in <71ed3aa6fd0e4ec4934b502e9e6ebc99>:0 

Vacuum command is taking 24+ hours
#6

Ok, so last time I tried the autocleanup flag, I forgot to add the =true part. So I tried again. I also logged profiling output to a logfile that is now using up a few gb of disk space…

My sanitized command:

sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe backup "ftp://<blah>" /home/user/ /etc/hosts /var/lib/mysql/ /opt/databasedumps/ --backup-name="BackupName" --dbpath=/root/.config/Duplicati/BackupName.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --keep-time=3Y --passphrase=<pass> <email options> --zip-compression-level=9 --run-script-timeout=3h --exclude-files-attributes="system,temporary" --disable-module=console-password-input <--exclude flags> --auto-cleanup=true log-level=Profiling log-file=/home/user/2018.05.10.duplicati.log

The last bit of the log file:

<lots of stuff>
2018-05-11 03:52:18Z - Profiling: ExecuteNonQuery: INSERT INTO "DeletedBlock" ("Hash", "Size", "VolumeID") SELECT "Hash", "Size", "VolumeID" FROM "Block" WHERE "ID" NOT IN (SELECT DISTINCT "BlockID" AS "BlockID" FROM "BlocksetEntry" UNION SELECT DISTINCT "ID" FROM "Block", "BlocklistHash" WHERE "Block"."Hash" = "BlocklistHash"."Hash")  took 00:00:21.171
2018-05-11 03:52:18Z - Profiling: Starting - ExecuteNonQuery: DELETE FROM "Block" WHERE "ID" NOT IN (SELECT DISTINCT "BlockID" FROM "BlocksetEntry" UNION SELECT DISTINCT "ID" FROM "Block", "BlocklistHash" WHERE "Block"."Hash" = "BlocklistHash"."Hash") 
2018-05-11 03:52:40Z - Profiling: ExecuteNonQuery: DELETE FROM "Block" WHERE "ID" NOT IN (SELECT DISTINCT "BlockID" FROM "BlocksetEntry" UNION SELECT DISTINCT "ID" FROM "Block", "BlocklistHash" WHERE "Block"."Hash" = "BlocklistHash"."Hash")  took 00:00:22.519
2018-05-11 03:52:40Z - Profiling: Starting - ExecuteNonQuery: UPDATE "RemoteVolume" SET "State" = ? WHERE "Type" = ? AND "State" IN (?, ?) AND "ID" NOT IN (SELECT "VolumeID" FROM "Fileset") 
2018-05-11 03:52:40Z - Profiling: ExecuteNonQuery: UPDATE "RemoteVolume" SET "State" = ? WHERE "Type" = ? AND "State" IN (?, ?) AND "ID" NOT IN (SELECT "VolumeID" FROM "Fileset")  took 00:00:00.019
2018-05-11 03:52:40Z - Error: Fatal error
System.Exception: Unexpected number of remote volumes marked as deleted. Found 0 filesets, but 1 volumes
  at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x00252] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0005e] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00169] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.CompactIfRequired (Duplicati.Library.Main.BackendManager backend, System.Int64 lastVolumeSize) [0x000a5] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter) [0x0076f] in <ae134c5a9abb455eb7f06c134d211773>:0 

2018-05-11 03:52:41Z - Profiling: Running Backup took 03:53:45.181
<stuff dealing with emailing me about the failed job.>

#7

I think if a boolean parameter is added without a value it defaults to true (meaning --auto-cleanup is the same as --auto-cleanup=true.

As far as the “Unexpected number of remote volumes marked as deleted” error goes our best bet is to try again to get @kenkendk involved.

Since you’re the third person (along with @glaurent and @tacioandrade) I know of to report this so far it may be an issue with some database state while updating to / beyond 2.0.3.3.


#8

Yes. All boolean options are false by default, so supplying them will set them to true, unless the value is either false, no, 0 or off.

The error is a fail-safe check that tests that the database query performs as expected, which is somehow not the case here.

It looks like it is trying to delete the oldest backup and this causes some trouble.

You can try:

sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe list "ftp://<blah>" --dbpath=/root/.config/Duplicati/BackupName.sqlite

This will show you all the versions Duplicati can find. You can then try to delete the oldest one like this (replace 8 with the right version from the list command):

sudo mono /usr/lib/duplicati/Duplicati.CommandLine.exe delete "ftp://<blah>" --dbpath=/root/.config/Duplicati/BackupName.sqlite 8

Maybe it works better when deleted manually.


#9

Er, I have it set to keep backups for 3 years. Why would it be trying to delete anything when it hasn’t been 3 years since I started using Duplicati?

In any case, I did try deleting the oldest fileset. This is what I got:

mono /usr/lib/duplicati/Duplicati.CommandLine.exe delete "ftp://..." --dbpath=/root/.config/Duplicati/blah.sqlite --passphrase=password --version=28
  Listing remote folder ...
promoting uploaded complete file from Uploading to Uploaded: duplicati-20180510T235856Z.dlist.zip.aes
Deleting 1 remote fileset(s) ...

System.Exception: Unexpected number of remote volumes marked as deleted. Found 1 filesets, but 2 volumes
  at Duplicati.Library.Main.Database.LocalDeleteDatabase+<DropFilesetsFromTable>d__5.MoveNext () [0x00252] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at System.Collections.Generic.LargeArrayBuilder`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] items) [0x0005e] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at System.Collections.Generic.EnumerableHelpers.ToArray[T] (System.Collections.Generic.IEnumerable`1[T] source) [0x00033] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at System.Linq.Enumerable.ToArray[TSource] (System.Collections.Generic.IEnumerable`1[T] source) [0x00018] in <c0502b9f220448589f6c0bfcd98a6c9b>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.DoRun (Duplicati.Library.Main.Database.LocalDeleteDatabase db, System.Data.IDbTransaction& transaction, System.Boolean hasVerifiedBacked, System.Boolean forceCompact, Duplicati.Library.Main.BackendManager sharedManager) [0x00169] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Operation.DeleteHandler.Run () [0x00070] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.<Delete>b__20_0 (Duplicati.Library.Main.DeleteResults result) [0x00012] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0014b] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.Action`1[T] method) [0x00009] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.Library.Main.Controller.Delete () [0x00006] in <ae134c5a9abb455eb7f06c134d211773>:0 
  at Duplicati.CommandLine.Commands.Delete (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Collections.Generic.List`1[T] args, System.Collections.Generic.Dictionary`2[TKey,TValue] options, Duplicati.Library.Utility.IFilter filter) [0x00087] in <71ed3aa6fd0e4ec4934b502e9e6ebc99>:0 
  at (wrapper delegate-invoke) System.Func`6[System.IO.TextWriter,System.Action`1[Duplicati.Library.Main.Controller],System.Collections.Generic.List`1[System.String],System.Collections.Generic.Dictionary`2[System.String,System.String],Duplicati.Library.Utility.IFilter,System.Int32].invoke_TResult_T1_T2_T3_T4_T5(System.IO.TextWriter,System.Action`1<Duplicati.Library.Main.Controller>,System.Collections.Generic.List`1<string>,System.Collections.Generic.Dictionary`2<string, string>,Duplicati.Library.Utility.IFilter)
  at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x003fb] in <71ed3aa6fd0e4ec4934b502e9e6ebc99>:0 

Is there a way to list the volumes that are marked as deleted, then compare that value to what’s really there and fix it manually somehow?


#10

Good question, it should not delete things then, but the error message indicates that this is what is happening. You should see a message like Deleting 1 remote filesets in the log for the backup.

My guess is that somehow a remote volume is marked as deleted incorrectly, and then all the database queries pick up this problem.

If you can make a bugreport for the database, I can have a look and see if I can fix it, but it will be a week or two before I can look.