Errors: please verify the sha256 hash

New user of Duplicati and former user of CrashPlan. I’m on Xubuntu Linux 18.04 with Duplicati 2.0.4.15_canary_2019-02-06.
After a little adjustment here and there, I got Duplicati backing up my Linux files to a Windows 10 share. Just recently I started getting warnings like this:

  • 2019-02-23 12:21:27 -06 - [Warning-Duplicati.Library.Main.Operation.FilelistProcessor-MissingRemoteHash]: remote file duplicati-b471d292a4c20498a9b6504cb85de762e.dblock.zip.aes is listed as Verified with size 10485760 but should be 52364077, please verify the sha256 hash “SlrV6tpAozmL7zKZjfMlI1Ntakoziu/0FLDgIvcdYXo=”

I ran the Commandline “list-broken-files” and it listed them. I then went back and did a “purge-broken-files” but all it did was list them again.

  1. What happened to cause these warnings to start appearing?
  2. Why doesn’t the “purge-broken-files” commandline remove the files in question? And will a subsequent backup attempt put the purged files back with the correct hash?

I’m hoping this is something simple because this is as close to Crashplan as I could find.

Is your Windows share mounted via CIFS/Samba? Apparently there are some issues with CIFS mounts and caching/truncation that can cause problems.

If you are using CIFS, can you try disabling caching to see if that helps? I don’t think it will fix the hash mismatch you mentioned, but hopefully it will prevent it from happening again in future backups.

See the below for possibly related issues:

Thanks for the suggestion. I am using CIFS/Samba so I added the cache=none to my fstab.
How do I clean up the hash problems? Should I just delete the backup and start over?

Though @warwickmm has been handling this well (thanks!), I’ve been having the urge to ask whether the wrong-sized file originally mentioned is the same wrong size when viewed from Windows, and also what it looks like when viewed in a hex editor (if you’re into that sort of thing) on either system. We more often see files get truncated to even binary values, and I suspect file system issues. They like to use binary values… Here, though, you had one seemingly expand from (in hex) 31F032D to A00000 which begs the question of whether you got the start of a good file plus extra stuff at its end (somehow). Extra stuff can be removed…

EDIT: Miscounted digits. Never mind the expansion, yours did the usual binary-even shrink, but it might still be good to look at Windows’s view and to see if at least the start looks right. Read the rest of this lightly…

AES File Format describes the file format, and gives a hex-viewer example. If you see AES, look at what’s around hex 31F032D, especially after that, although if done right, that should have already been end-of-file.

You can also see if you can find that file’s original upload size and hash, e.g. Job → Show log → Remote then if you can find the put of that file, click on it to find the file size and hash that were originally recorded.

I took one of the flagged files (duplicati-b471d292a4c20498a9b6504cb85de762e.dblock.zip.aes) and found it in the remote server. I tried to jump to 31F032D using a hex editor but there is no such address.
I also looked in the Remote log and there is no “put” entry for this particular file. My warning log says there are 230 entries, but the Remote log doesn’t show this many - it shows only 5 messages.
Note that this went from working one day to showing these warnings the next. Nothing changed on either end.
I’m going to try deleting the whole backup and starting over.

I tried to start over and got some warnings on some directories so I excluded them. Now I’m down to 2 warnings.

I suppose I could also exclude the “gvfs-metadata/home” directory, but what’s up with the SQL file error?

Starting to think Duplicati is not for me, but I’ll work with it a bit longer.

  • 2019-02-25 07:46:04 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /home/user/.local/share/gvfs-metadata/home
  • 2019-02-25 07:47:45 -06 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: /home/user/.config/Duplicati/71868281878084798189.sqlite-journal

Please use the Database option for your job to see if the Local database path is the same as in the warning except for lacking a -journal suffix. If so, this is the job’s database rollback journal which is a temporary file which comes and goes and probably tends to be in use (i.e. locked) when it’s there at all.

If you want to see the details of the error (not visible in the summary), you can watch the server log, i.e. About → Show log → Live → Warning, or set a –log-file for the job if you think you might want it again.

Generally, you don’t want a job to back up its own database. Even if not locked, it’s changing, so is stale instantly in terms of content. It tracks the backup which is running. Back it up in a different job, if you like.

For an example of a locked file error in detail, here’s a live log where I click on the warning to see details:

Why doesn’t Duplicati skip backing up it’s own database file if it causes problems?

I can’t speak for developers. To me this is a rare and new “nuisance” warning, not a technical problem, so my guess is it’s too new, does too little damage, and has a workaround. Many other problem do not. More:

From a philosophy point of view, Duplicati seems to me (and I am just a user) to be one of those programs that does not force a lot of restrictions on the user, but gives them large latitude to do the backup they want. Some other backup programs take an opposite approach, taking responsibility and capability from the user.

From a problem point of view, this appears to have only become a (rare) warning in the recent 2.0.4.5 beta (with some found in canary releases). It’s often near a #1400 error. Both might be 2.0.3.6 new-code issues.

New warnings/errors occurring after updating to 2.0.4.5

Errors are more serious than warnings, and there are enough still lingering that rare new warnings are not likely high on the priority list for scarce developer time. From a technical point of view, removing this could mean making filters more dynamic, because the database name changes. Current filters appear fixed, but do at least customize several paths to the user. Database path varies and can’t be shown in format below:

C:\Program Files\Duplicati 2>Duplicati.CommandLine.exe help filter-groups

Duplicati has several built-in groups of filters, which include commonly
excluded files and folders for different operating systems. These sets can be
specified via the --include/--exclude parameter by putting their name in
{curly brackets}.

    None: Selects no filters.
    DefaultExcludes: A set of default exclude filters, currently evaluates to:
    DefaultIncludes, SystemFiles,DefaultIncludes,
    OperatingSystem,DefaultIncludes, CacheFiles,DefaultIncludes,
    TemporaryFiles,DefaultIncludes, Applications.
    DefaultIncludes: A set of default include filters, currently evaluates to:
    None.

    SystemFiles: Files that are owned by the system or not suited to be backed
    up. This includes any operating system reported protected files. Most
    users should at least apply these filters.
     Aliases: Sys,System
      *UsrClass.dat*
      *\I386*
      *\Internet Explorer\
      *\Microsoft*\RecoveryStore*
      *\Microsoft*\Windows\*.edb
      *\Microsoft*\Windows\*.log
      *\Microsoft*\Windows\Cookies*
      *\MSOCache*
      *\NTUSER*
      *\ntuser.dat*
      *\RECYCLER\
      ?:\hiberfil.sys
      ?:\pagefile.sys
      ?:\swapfile.sys
      ?:\System Volume Information\
      ?:\Windows\Installer*
      ?:\Windows\Temp*
      C:\ProgramData\Microsoft\Network\Downloader\*
      C:\ProgramData\Microsoft\Windows\WER\*
      C:\Users\xxx\AppData\Local\Temp\*
      C:\Users\xxx\index.dat
      C:\WINDOWS\memory.dmp
      C:\WINDOWS\Minidump\*
      C:\WINDOWS\netlogon.chg
      C:\WINDOWS\softwaredistribution\*.*
      C:\WINDOWS\system32\LogFiles\WMI\RtBackup\*.*
      C:\Windows\system32\MSDtc\MSDTC.LOG
      C:\Windows\system32\MSDtc\trace\dtctrace.log

    OperatingSystem: Files that belong to the operating system. These files
    are restored when the operating system is re-installed.
     Aliases: OS
      *\Recent\
      ?:\autoexec.bat
      ?:\Config.Msi*
      C:\Users\xxx\AppData\Roaming\Microsoft\Windows\Recent\
      C:\WINDOWS.old\
      C:\WINDOWS\

    TemporaryFiles: Files and folders that are known to be storage of
    temporary data.
     Aliases: Temp,TempFiles,TempFolders,TemporaryFolders,Tmp
      *.tmp
      *.tmp\
      *\$RECYCLE.BIN\
      *\AppData\Local\Temp*
      *\AppData\Temp*
      *\Local Settings\Temp*
      ?:\Windows\Temp*
      C:\Users\xxx\AppData\Local\Temp\

    CacheFiles: Files and folders that are known cache locations for the
    operating system and various applications
     Aliases: Cache,CacheFolders,Caches
      *\AppData\Local\AMD\DxCache\
      *\AppData\Local\Apple Computer\Mobile Sync\
      *\AppData\Local\Comms\UnistoreDB\
      *\AppData\Local\ElevatedDiagnostics\
      *\AppData\Local\Microsoft\VSCommon\*SQM*
      *\AppData\Local\Microsoft\Windows Store\
      *\AppData\Local\Microsoft\Windows\Explorer\
      *\AppData\Local\Microsoft\Windows\INetCache\
      *\AppData\Local\Microsoft\Windows\UPPS\
      *\AppData\Local\Microsoft\Windows\WebCache*
      *\AppData\Local\Packages\
      *\Application Data\Apple Computer\Mobile Sync\
      *\Application Data\Application Data*
      *\Dropbox\Dropbox.exe.log
      *\Dropbox\QuitReports\
      *\Duplicati\control_dir_v2\
      *\Google\Chrome\*cache*
      *\Google\Chrome\*Current*
      *\Google\Chrome\*LOCK*
      *\Google\Chrome\Safe Browsing*
      *\Google\Chrome\User Data\Default\Cookies
      *\Google\Chrome\User Data\Default\Cookies-journal
      *\iPhoto Library\iPod Photo Cache\
      *\Local Settings\History\
      *\Mozilla\Firefox\*cache*
      *\OneDrive\.849C9593-D756-4E56-8D6E-42412F2A707B
      *\Safari\Library\Caches\
      *\Temporary Internet Files\
      *\Thumbs.db
      C:\Users\xxx\AppData\Local\Microsoft\Windows\History\
      C:\Users\xxx\AppData\Local\Microsoft\Windows\INetCache\
      C:\Users\xxx\AppData\Local\Microsoft\Windows\INetCookies\
      [.*\\(cookies|permissions).sqllite(-.{3})?]

    Applications: Installed programs and their libraries, but not their
    settings.
     Aliases: App,Application,Apps
      C:\Program Files (x86)\
      C:\Program Files\
      C:\ProgramData\Microsoft\Windows\Start Menu\Programs\Administrative
      Tools\
      C:\Users\xxx\AppData\Roaming\Microsoft\Windows\Start
      Menu\Programs\Administrative Tools\




C:\Program Files\Duplicati 2>

Filtering has been found to cause quite a large loss of performance, but that’s been improved in 2.0.4.10:

Improved performance of filters by around 10x, thanks @warwickmm

I’m not sure how much performance loss filtering out the job’s own database would cause. If need be, the removal could perhaps be done by special-casing comparison instead of the usual filtering mechanism…

It seems a reasonable idea to me for new users, but if there are users who now backup their database to the extent that backup is possible, these users might be hugely surprised if Duplicati suddenly stops that.

To me, this has more the flavor of an issue than a new Feature, but you can make a case for it either way, or possibly someone will read this discussion. Ultimately, some volunteer has to decide to go off and do it.