I’ve never been able to sustain Duplicati backups very long because I always get an “unexpected difference in fileset” error after a few days that blocks further backups. It appears that the error is always (or almost always) just a difference of one file. Is there any way to determine why this keeps happening, what the missing file is, how to prevent/fix it?
The only suggestion I’ve seen in this forum is to delete the bad fileset. I can do this, although I takes several hours. I recently had this error for fileset version 3, so deleted that, but before I could do a successful backup I got the error for fileset version 1. If I delete 1 and then have to delete 0, don’t I then lose all my base image so my next incremental backup is basically everything?
Also, is the “ListBrokenFiles” command related to this issue? I thought I would try that on the broken version (after a repair had no effect), and got this output/error:
Input command: list-broken-files
Input arguments:
file:///[REDACTED]
Input options:
backup-name: [REDACTED]
dbpath: [REDACTED]
encryption-module: aes
compression-module: zip
dblock-size: 50mb
passphrase: [REDACTED]
retention-policy: 1W:1D,4W:1W,12M:1M
skip-files-larger-than: 15GB
console-log-level: profiling
debug-output: true
version: 1
The operation ListBrokenFiles has started
Starting - Running ListBrokenFiles
Starting - ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES ("ListBrokenFiles", 1551389697); SELECT last_insert_rowid();
ExecuteScalarInt64: INSERT INTO "Operation" ("Description", "Timestamp") VALUES ("ListBrokenFiles", 1551389697); SELECT last_insert_rowid(); took 0:00:00:00.083
Starting - ExecuteReader: SELECT "Key", "Value" FROM "Configuration"
ExecuteReader: SELECT "Key", "Value" FROM "Configuration" took 0:00:00:00.000
Starting - ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC
ExecuteReader: SELECT "ID", "Timestamp" FROM "Fileset" ORDER BY "Timestamp" DESC took 0:00:00:00.000
Starting - ExecuteReader: SELECT DISTINCT "B"."Timestamp", "A"."FilesetID", COUNT("A"."FileID") AS "FileCount" FROM "FilesetEntry" A, "Fileset" B WHERE "A"."FilesetID" = "B"."ID" AND "A"."FileID" IN (
SELECT DISTINCT "ID" FROM (
SELECT "ID" AS "ID", "BlocksetID" AS "BlocksetID" FROM "FileLookup" WHERE "BlocksetID" != -100 AND "BlocksetID" != -200
UNION
SELECT "A"."ID" AS "ID", "B"."BlocksetID" AS "BlocksetID" FROM "FileLookup" A LEFT JOIN "Metadataset" B ON "A"."MetadataID" = "B"."ID"
)
WHERE "BlocksetID" IS NULL OR "BlocksetID" IN
(
SELECT DISTINCT "BlocksetID" FROM
(
SELECT "BlocksetID" FROM "BlocksetEntry" WHERE "BlockID" NOT IN
(SELECT "ID" FROM "Block" WHERE "VolumeID" IN
(SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
UNION
SELECT "BlocksetID" FROM "BlocklistHash" WHERE "Hash" NOT IN
(SELECT "Hash" FROM "Block" WHERE "VolumeID" IN
(SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
UNION
SELECT "A"."ID" AS "BlocksetID" FROM "Blockset" A LEFT JOIN "BlocksetEntry" B ON "A"."ID" = "B"."BlocksetID" WHERE "A"."Length" > 0 AND "B"."BlocksetID" IS NULL
)
WHERE "BlocksetID" NOT IN (SELECT "ID" FROM "Blockset" WHERE "Length" == 0)
)
) AND "A"."FilesetID" IN (SELECT "ID" FROM "Fileset" WHERE "ID" IN (58)) GROUP BY "A"."FilesetID"
ExecuteReader: SELECT DISTINCT "B"."Timestamp", "A"."FilesetID", COUNT("A"."FileID") AS "FileCount" FROM "FilesetEntry" A, "Fileset" B WHERE "A"."FilesetID" = "B"."ID" AND "A"."FileID" IN (
SELECT DISTINCT "ID" FROM (
SELECT "ID" AS "ID", "BlocksetID" AS "BlocksetID" FROM "FileLookup" WHERE "BlocksetID" != -100 AND "BlocksetID" != -200
UNION
SELECT "A"."ID" AS "ID", "B"."BlocksetID" AS "BlocksetID" FROM "FileLookup" A LEFT JOIN "Metadataset" B ON "A"."MetadataID" = "B"."ID"
)
WHERE "BlocksetID" IS NULL OR "BlocksetID" IN
(
SELECT DISTINCT "BlocksetID" FROM
(
SELECT "BlocksetID" FROM "BlocksetEntry" WHERE "BlockID" NOT IN
(SELECT "ID" FROM "Block" WHERE "VolumeID" IN
(SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
UNION
SELECT "BlocksetID" FROM "BlocklistHash" WHERE "Hash" NOT IN
(SELECT "Hash" FROM "Block" WHERE "VolumeID" IN
(SELECT "ID" FROM "RemoteVolume" WHERE "Type" = "Blocks"))
UNION
SELECT "A"."ID" AS "BlocksetID" FROM "Blockset" A LEFT JOIN "BlocksetEntry" B ON "A"."ID" = "B"."BlocksetID" WHERE "A"."Length" > 0 AND "B"."BlocksetID" IS NULL
)
WHERE "BlocksetID" NOT IN (SELECT "ID" FROM "Blockset" WHERE "Length" == 0)
)
) AND "A"."FilesetID" IN (SELECT "ID" FROM "Fileset" WHERE "ID" IN (58)) GROUP BY "A"."FilesetID" took 0:00:18:44.361
Running ListBrokenFiles took 0:00:18:45.347
Mono.Data.Sqlite.SqliteException (0x80004005): SQLite error
cannot rollback - no transaction is active
at Mono.Data.Sqlite.SQLite3.Reset (Mono.Data.Sqlite.SqliteStatement stmt) [0x00096] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SQLite3.Step (Mono.Data.Sqlite.SqliteStatement stmt) [0x00046] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteDataReader.NextResult () [0x00129] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteDataReader..ctor (Mono.Data.Sqlite.SqliteCommand cmd, System.Data.CommandBehavior behave) [0x00051] in <fe9fd999cd9f407db94500dce293e66f>:0
at (wrapper remoting-invoke-with-check) Mono.Data.Sqlite.SqliteDataReader:.ctor (Mono.Data.Sqlite.SqliteCommand,System.Data.CommandBehavior)
at Mono.Data.Sqlite.SqliteCommand.ExecuteReader (System.Data.CommandBehavior behavior) [0x00006] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteCommand.ExecuteNonQuery () [0x00000] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteTransaction.IssueRollback (Mono.Data.Sqlite.SqliteConnection cnn) [0x00012] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteTransaction.Rollback () [0x00008] in <fe9fd999cd9f407db94500dce293e66f>:0
at Mono.Data.Sqlite.SqliteTransaction.Dispose (System.Boolean disposing) [0x0001e] in <fe9fd999cd9f407db94500dce293e66f>:0
at System.Data.Common.DbTransaction.Dispose () [0x00000] in <a566c189d19d4271ab58420d6f875b13>:0
at Duplicati.Library.Main.Operation.ListBrokenFilesHandler.Run (Duplicati.Library.Utility.IFilter filter, System.Func`6[T1,T2,T3,T4,T5,TResult] callbackhandler) [0x00059] in <fbbeda0cad134e648d781c1357bdba9c>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass33_0.<ListBrokenFiles>b__0 (Duplicati.Library.Main.ListBrokenFilesResults result) [0x0001c] in <fbbeda0cad134e648d781c1357bdba9c>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011d] in <fbbeda0cad134e648d781c1357bdba9c>:0
