,Duplicati.Library.Utility.IFilter)
at Duplicati.CommandLine.Program.ParseCommandLine (System.IO.TextWriter outwriter, System.Action`1[T] setup, System.Boolean& verboseErrors, System.String[] args) [0x00342] in <3d18b3352444419eba6d6a9e7e773335>:0
at Duplicati.CommandLine.Program.RunCommandLine (System.IO.TextWriter outwriter, System.IO.TextWriter errwriter, System.Action`1[T] setup, System.String[] args) [0x00002] in <3d18b3352444419eba6d6a9e7e773335>:0
```
The command line used is this (redacted) one:
```
/usr/bin/mono /opt/duplicati-latest/Duplicati.CommandLine.exe backup --backup-name="Full backup" --dbpath=/var/lib/duplicati/.config/Duplicati/MYDATABASE.sqlite --encryption-module=aes --compression-module=zip --dblock-size=50mb --passphrase="PASS PHRASE" --disable-module=console-password-input --asynchronous-concurrent-upload-limit=10 --retention-policy="1W:1D,4W:1W,12M:1M" --exclude="*lost+found*/" --exclude="*/a/path/to/exclude*/" "b2://bucket-name/?auth-username=userid&auth-password=userpassword" /path/to/backup/
```
As you can see from the path used, this is running on a linux machine, using the Manjaro distribution and the package from the AUR. I’m mentionning it just in case, but I don’t think it has any impact here.
I really appreciate the time you are spending to help me on this.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [January 1, 2022, 2:20am UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/5 "2022-01-01T02:20:08Z")
> [@obones](#):
>
> it appears that the report occurs not long after “compact” has been attempted.
Maybe you have had different ones. The issue on download could be in compact which only runs when needed, and only compacts files with wasted space. Or it could be in sampled test of files after backup.
> [@obones](#):
>
> ```plaintext
> Backend event: Get - Failed: duplicati-b71d0b311be324f5e90c0ad1e5f091b5e.dblock.zip.aes (49,97 MB)
>
> ```
So at least duplicati-b71d0b311be324f5e90c0ad1e5f091b5e.dblock.zip.aes is having trouble. B2 doesn’t have a great web UI, but I wonder if you could get a timestamp of the file, and a download to try to check although there’s a limited amount possible with an encrypted file because most should look very random.
In addition to timestamp, it might be helpful to know where the file uploaded in the backup uploads series. You can infer this from timestamps if you can find a B2 client that can sort by date. On Windows, maybe Cyberduck would do. I’m not sure what there is for Linux. Another way to find the file’s history is in the DB using `sqlitebrowser` which I think is in your repo. I could guide you. It looks a bit like spreadsheet tables.
Posting a link to a [database bug report](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#creating-a-bug-report) would be another way for somebody to look for clues about the file.
---
### Author:  [@obones](https://forum.duplicati.com/u/obones)
#### Post date: [January 1, 2022, 11:18am UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/6 "2022-01-01T11:18:34Z")
> [@ts678](#):
>
> I wonder if you could get a timestamp of the file, and a download to try to check
I already had Cyberduck installed on a Windows workstation, so it was quite easy to get to that file. Its timestamp is set to 2021-08-27 04.:30:55 which sounds right for the time part as the backup script runs around that time of night.
I downloaded it and it has this at its beginning:
So it looks like at least the header is the expected one.
> [@ts678](#):
>
> it might be helpful to know where the file uploaded in the backup uploads series
This is what I get:
> [@ts678](#):
>
> Another way to find the file’s history is in the DB
Well, sqlite3 is something I’m used to, here is the query I ran:
```plaintext
sqlite> select * from Remotevolume where Name="duplicati-b71d0b311be324f5e90c0ad1e5f091b5e.dblock.zip.aes" limit 10;
20127|96|duplicati-b71d0b311be324f5e90c0ad1e5f091b5e.dblock.zip.aes|Blocks|52393629|R5o1CiSclJjMWDPCxKVVTzwn+QUj33b8XdHgO+4B0Yg=|Verified|0|0
```
But I don’t know the schema well enough to infer anything from this, nor to find the appropriate joins that I could do to get more information.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [January 1, 2022, 1:00pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/7 "2022-01-01T13:00:34Z")
> [@obones](#):
>
> This is what I get:
Is there genuinely no `dlist` file from the Aug 27 backup? That uploads at end of backup. Sample activity:
Yours seems to be doing about the right pairs of dblock and dindex, then it stops. I wonder what that is?
Does your Remotevolume table think that there’s a dlist file associated with the backup (like as above)?
A `dlist` represents a Fileset. You can look for a fileset. Is `IsFullBackup` a 1? Maybe it was interrupted.
Is there a log entry made? This is easier to see in a GUI, but GUI is just displaying what database holds:
> [@obones](#):
>
> `|Verified|0|`
Do you have lots looking like that? I seem to have 20 of 514 that the small default verify hasn’t [sampled](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#verifying-backend-files).
Sampling can be raised if the problem continues, so at least it may warn earlier rather than in compact.
> [@obones](#):
>
> So it looks like at least the header is the expected one.
At least some header errors get a different error message. In spite of the runs of 0-filled lines, you can
`hexdump -C` or `od -c` it and pipe to `grep '^*$` to see if there are big empty blocks later in encryption.
Checking the file size against DB size might be good, but I think Duplicati is already checking file size.
File content, unfortunately, can’t be checked without downloading the file and seeing if something fails.
[AES Crypt](https://www.aescrypt.com/) or Duplicati’s CLI SharpAESCrypt.exe can be used on that file to confirm that it’s damaged.
---
### Author:  [@obones](https://forum.duplicati.com/u/obones)
#### Post date: [January 1, 2022, 4:47pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/8 "2022-01-01T16:47:23Z")
> [@ts678](#):
>
> Is there genuinely no `dlist` file from the Aug 27 backup?
No, nor are there many `dlist` files. In the database, there are only 14 of them, all verified but the one from last night.
> [@ts678](#):
>
> Does your Remotevolume table think that there’s a dlist file associated with the backup (like as above)?
No `Files` type for that operation. What’s strange though, is that `RemoteOperation` contains data starting December 16, nothing earlier than that.
What I mean is that `Operation` contains IDs starting at 1 as expected, dated on May 1st 2021, but none of these operations are to be found in `RemoteOperation`
The `RemoteVolume` table has those operations however.
> [@ts678](#):
>
> Is `IsFullBackup` a 1?
The `Fileset` table only contains 14 rows, as expected from the earlier lookup in `RemoteVolume`, none which are linked to the operation of interest (96)
In that table, only 4 have `IsFullBackup` set to 0:
> [@ts678](#):
>
> Is there a log entry made?
Alas not, only 6 records in that table, and all for very recent operations, just like what can be seen in the `RemoteOperation` table, it starts at `OperationID` 226 from December 16th.
> [@ts678](#):
>
> Do you have lots looking like that?
Well, I guess I can say yes:
```plaintext
sqlite> select Count(*) from Remotevolume where VerificationCount=0;
20149
sqlite> select Count(*) from Remotevolume;
20384
```
> [@ts678](#):
>
> [AES Crypt](https://www.aescrypt.com/) or Duplicati’s CLI SharpAESCrypt.exe can be used on that file to confirm that it’s damaged.
I used CyberDuck to download the file and ran `aescrypt -e -p "PASS PHRASE"` on it and got this:
```
Error: Message has been altered and should not be trusted
```
Same error message as Duplicati, which is definitely not surprising.
I hope this clarifies the situation a bit
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [January 1, 2022, 6:47pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/9 "2022-01-01T18:47:12Z")
> [@obones](#):
>
> there are only 14 of them
That’s not a guaranteed issue. I think the count should match Fileset row count and the Remotevolume Type Files count (ignoring State Deleted). I have 21 expected dlist, which matches my home page’s “21 Versions”.
> [@obones](#):
>
> contains data starting December 16, nothing earlier than that.
I forgot about [log-retention](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#log-retention) which will interfere with this and LogData, but default is 30 days, unless changed. Another way to lose old data of this sort is to recreate the database. That gets essentials but not the log info.
Operation table does not seem to be trimmed. Mine goes to Oct 6, but the two trimmed ones go for 30 days.
> [@obones](#):
>
> none of these operations are to be found in `RemoteOperation`
It might be the RemoteOperation table trimming, but retention trim seems too soon unless you decreased it. Lack of such trimming is another reason why an external [log-file](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#log-file) is best. It also shows what DB rolls back…
There are some issues where, for example, a compact deletes a dindex, errors out, and rolls back DB data. Actual destination file deletion remains, of course, so next backup thinks it has a missing file, but I digress…
Back to the `dlist`, your retention policy deletes backup versions, so maybe that’s why there’s no aged dlist. External log at even [log-file-log-level](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#log-file-log-level)=Information is enough to show file deletions. Retry level is a little better.
> [@obones](#):
>
> In that table, only 4 have `IsFullBackup` set to 0:
0 is not exactly an all-went-fine result, but can occur if backup is interrupted. It would list as partial in the GUI.
Converting some Timestamp to UTC, it looks like the Aug 27 backup got deleted by retention. Gap in Fileset:
```plaintext
1629252391 August 18, 2021
1632276432 September 22, 2021
```
> [@obones](#):
>
> it starts at `OperationID` 226 from December 16th.
Do you by any chance have log-retention set to `14D`?
> [@obones](#):
>
> `VerificationCount=0`
To have fewer of these (and more verification, and unfortunately more B2 download charges), you can use [backup-test-samples](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#backup-test-samples) or [backup-test-percentage](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#backup-test-percentage) to keep a closer eye on the contents of the uploaded files.
[The TEST command](https://duplicati.readthedocs.io/en/latest/04-using-duplicati-from-the-command-line/#the-test-command) from [Commandline](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#using-the-command-line-tools-from-within-the-graphical-user-interface) or shell can also test-on-demand as many files as you would like.
I was hoping that more could be gotten from the DB, but there’s too much missing. The damaged dblock is uploaded at end of backup, and even though the dindex files were later, the dblock files might have finished after the dindex did (because they’re larger). I don’t know if future inspection will show issue-at-end pattern.
You can stop file upload from finishing out-of-sequence by setting this option to 1. It may slow upload rates:
```plaintext
--asynchronous-concurrent-upload-limit (Integer): The number of concurrent
uploads allowed
When performing asynchronous uploads, the maximum number of concurrent
uploads allowed. Set to zero to disable the limit.
* default value: 4
```
Free options include increasing log retention, setting up external log file, playing with upload concurrency, looking at damaged files for any interesting patterns (e.g. empty blocks in the middle or at the end of file).
You might be able to correlate errors with something. For example, you might find system boots like this:
```plaintext
who --boot /var/log/wtmp
system boot 2021-01-31 19:42
system boot 2021-01-31 20:13
system boot 2021-10-08 09:45
system boot 2021-10-08 10:26
system boot 2021-11-30 15:14
```
> [@obones](#):
>
> Well, sqlite3 is something I’m used to,
Would you consider helping any with either SQL queries or issues that might flat-out be SQLite. Scaling to large backups has been a problem, and I’ve seemingly had DB Browser for SQLite read DB for [26 days](https://github.com/duplicati/duplicati/issues/4645#issuecomment-1000373099)…
There’s other technical info there, and some concerning reports about SQLite performance dropoffs [here](https://forum.duplicati.com/t/why-the-heck-cant-we-change-the-blocksize/13120/69).
---
### Author:  [@obones](https://forum.duplicati.com/u/obones)
#### Post date: [January 4, 2022, 2:38pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/10 "2022-01-04T14:38:40Z")
> [@ts678](#):
>
> Do you by any chance have log-retention set to `14D`?
I do not specify it on the command line, so I believe it has its default value of `30D`. But the source data does not change not that often, so it’s completely possible nothing happened between mid November and mid December.
> [@ts678](#):
>
> [The TEST command](https://duplicati.readthedocs.io/en/latest/04-using-duplicati-from-the-command-line/#the-test-command) from [Commandline](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#using-the-command-line-tools-from-within-the-graphical-user-interface) or shell can also test-on-demand as many files as you would like.
Yes, I ran it on another backup that has a local hard drive as a target, and it worked just fine. Well, it complained about extras, but it appears these are benign and can be [ignored](https://forum.duplicati.com/t/verify-a-specific-dblock-zip-aes-file/3500/6)
> [@ts678](#):
>
> It may slow upload rates
Yes, it slows them to a crawl, B2 is throttling individual uploads.
> [@ts678](#):
>
> Would you consider helping any with either SQL queries or issues that might flat-out be SQLite
I can give a hand, yes, of course. That being said, I have never encountered an error in SQLite itself, but always in my own setup of the library, or the queries I run against it.
For the problem at hand, I have created an [issue](https://github.com/duplicati/duplicati/issues/4658) to get the name of the broken file even at default log level. Then I removed the broken file from the backend as it keeps popping up as an error, ran `list-broken-files` and `purge-broken-files` to, I hope, get to a fixed situation.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [January 5, 2022, 2:30am UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/11 "2022-01-05T02:30:17Z")
> [@obones](#):
>
> get the name of the broken file even at default log level.
I never lack a name when I get an error (testing on Windows to local folder though) but errors are erratic. Corrupting an encrypted dblock with Frhed to change some byte in the middle of the file gets hash error:
```plaintext
Running commandline entry
Finished!
Backup started at 1/4/2022 8:34:01 PM
Checking remote backup ...
Listing remote folder ...
Scanning local files ...
0 files need to be examined (0 bytes)
Uploading file (1.01 KB) ...
Checking remote backup ...
Listing remote folder ...
Verifying remote backup ...
Remote backup verification completed
Downloading file (1.01 KB) ...
Downloading file (27.22 KB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Failed to process file duplicati-b630d3bea562145a49486d0a5d8accffa.dblock.zip.aes => Hash mismatch on file "C:\Users\Maintenance\AppData\Local\Temp\dup-ed0bdaf9-a5b1-46bd-93ec-71145eb148d4", recorded hash: KoisxBN3gpcR5yD4nhXETCj3+I2yGX8oRvapbfjzSts=, actual hash 8tJuFk8ridaufnjn/4NX0V9xQg/cW/TWtnDs5MPSaZA=
Duration of backup: 00:00:42
Remote files: 4
Remote size: 39.30 MB
Total remote quota: 930.97 GB
Available remote quota: 279.01 GB
Files added: 0
Files deleted: 0
Files changed: 0
Data uploaded: 1.01 KB
Data downloaded: 196.37 MB
Backup completed successfully!
Return code: 3
```
To try to bypass that, I get the SHA-256 sum of the modified dblock (using DigitalVolcano Hash Tool), paste hex to [https://cryptii.com/](https://cryptii.com/) to convert to RFC 3548 Base64, and edit the result into the Remotevolume table.
This is good enough for the after-backup verification (which I tried to make more realistic by setting [upload-unchanged-backups](https://duplicati.readthedocs.io/en/latest/06-advanced-options/#upload-unchanged-backups)), but I can get a [Commandline](https://duplicati.readthedocs.io/en/latest/03-using-the-graphical-user-interface/#using-the-command-line-tools-from-within-the-graphical-user-interface) `test` to notice the corrupted file and fail but with name:
```plaintext
Running commandline entry
Finished!
Listing remote folder ...
Downloading file (1.01 KB) ...
Downloading file (27.22 KB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Downloading file (39.27 MB) ...
Failed to process file duplicati-b630d3bea562145a49486d0a5d8accffa.dblock.zip.aes => Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
duplicati-i6505dfd834f14302adc113c3a0cafc54.dindex.zip.aes: 1 errors
Modified: duplicati-b630d3bea562145a49486d0a5d8accffa.dblock.zip.aes
duplicati-b630d3bea562145a49486d0a5d8accffa.dblock.zip.aes: 1 errors
Error: Failed to decrypt data (invalid passphrase?): Message has been altered, do not trust content
Return code: 3
```
I’m not familiar with the code and wouldn’t mind if a developer were willing to help look into verification plan. Preferably they could try Linux or you might try Windows on `duplicati-2.0.6.3_beta_2021-06-17.zip` from [2.0.6.3\_beta\_2021-06-17](https://github.com/duplicati/duplicati/releases/tag/v2.0.6.3-2.0.6.3_beta_2021-06-17). Passphrase is `passphrase` and I corrupted offset 0x200 from 0xe9 to 0x00 although the exact contents, hash, etc. will probably vary. There’s a “Created” time field in its `manifest` file. [AES File Format](https://www.aescrypt.com/aes_file_format.html) has some header and trailer info that I wanted to miss – hit the middle of 1-dblock backup where default sample size of 1 is going to sample the corrupted dblock without my having to watch names.
---
### Author:  [@obones](https://forum.duplicati.com/u/obones)
#### Post date: [January 5, 2022, 4:40pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/12 "2022-01-05T16:40:51Z")
> [@ts678](#):
>
> I never lack a name when I get an error
Could it be because I’m using parallel upload? This way, the exceptions are “grouped” together in an `AggregateException` that may not get displayed entirely properly on the command line output.
---
### Author:  [@ts678](https://forum.duplicati.com/u/ts678)
#### Post date: [January 5, 2022, 5:56pm UTC](https://forum.duplicati.com/t/recovering-from-message-alteration/13638/13 "2022-01-05T17:56:25Z")
> [@obones](#):
>
> Could it be because I’m using parallel upload?
I’m using default of 4 concurrent, as well as default of pretty much everything else that wasn’t stated.
If you’ve got some non-default options, maybe something is doing it, or maybe it’s Linux. I don’t know.
I would hope that the uploads are done by the time file sampling starts, or the file might not be up yet.
Clearly something is behind the different results, but more testing is needed to try to isolate the issue.