I ran a test restore today of a subfolder (about 4GB of mixed files including images, text files, photoshop files, and video). I used the no local blocks option to be sure all files were recovered from the original backup destination (a USB hard drive).
At the end I received these errors (slightly obfuscated filenames):
Failed to restore file: “/mnt/nas/Restore/d_b/assets/FILE-front-360_0319_Stitch_XHC.psd”. File hash is 77OWUvuPkGtvxIE4Ula/XBJOg02bUMMx1Lw1AYWnEjE=, expected hash is 3HmjOeHxskwI38kH4LQosjVl1JV/SgqvMBjsoEL3CUE= => Failed to restore file: “/mnt/nas/Restore/d_b/assets/FILE-front-360_0319_Stitch_XHC.psd”. File hash is 77OWUvuPkGtvxIE4Ula/XBJOg02bUMMx1Lw1AYWnEjE=, expected hash is 3HmjOeHxskwI38kH4LQosjVl1JV/SgqvMBjsoEL3CUE=,
Failed to restore file: “/mnt/nas/Restore/d_b/images/events/FILE (2).jpg”. File hash is QfHbk4ZHjW5t3KBN/GDlmVE6f4AygfTBrjCwEYzqzhU=, expected hash is jIo3J6JmfVabtIz4YYIu2d72hnNBB6QAet/+InX6rhM= => Failed to restore file: “/mnt/nas/Restore/d_b/images/events/FILE (2).jpg”. File hash is QfHbk4ZHjW5t3KBN/GDlmVE6f4AygfTBrjCwEYzqzhU=, expected hash is jIo3J6JmfVabtIz4YYIu2d72hnNBB6QAet/+InX6rhM=
I’m wondering what I can do to troubleshoot this and is there a command that I can use to have Duplicati check the entire backup set to look for other errors? This is two files out of 4GB restore on a backup that is about 60 GB in total, so I am concerned there are other errors, though I haven’t run a restore on the entire backup.
The original files are OK (on the original source) and restored OK from my duplicati backup stored on Dropbox. The recovered files are definitely corrupted (the jpeg is clipped and photoshop reports that the PSD is corrupt). It appears the damage is limited to this destination, but it is still concerning to me.
Quick edit - this backup for this source and destination has never generated any errors or warnings since I set it up a few weeks ago.
Restoring from a USB drive (connected to the duplicati server) to my NAS over CIFS. The NAS/CIFS seems to be a never-ending source of issues for me. I am thinking of just replacing it.
Is that in general or just with Duplicati (I’ve seen some of your other posts).
I’m not really clear on the difference between / history of CIFS vs SAMBA but I’ve not had any issues with my (Linux based) unRAID box shares with various Windows machines.
Though admittedly my Duplicati backups (even internal LAN) are all done via SFTP…
I’ve only had issues (that I’m aware of) with Duplicati with respect to this NAS. We use it to store documents, photos, music, and movies, and we’ve never noticed any issues pulling files from it via streaming or Windows file transfer. I have TV shows saved on it that we watch in Sage over the LAN, we watch movies from it, etc. But I’m not making any statement against Duplicati there; I just don’t know what’s going on, but all the issues I’ve had backing up (and now restoring) seem to stem from that box and/or how my Ubuntu server is moving data to and from it.
I use FTPS for backups now too since I was having sha256 hash errors using CIFS. Everything has been fine on the backup end. It seems like there is some corruption happening either on the NAS or during the file transfer that makes my combination unreliable.
I’ve done two test restores to the root partition of the Duplicati drive and they’re both perfect, so the integrity of the backups appears to be fine.
I turned on NFS as a protocol on the NAS, mounted my restore share using NFS to a different mount point, and the restore worked perfectly. Then I went back and did another test restore to the CIFS share and it generated three errors on different files.
For anyone playing along, I learned something else interesting. I went back and re-tried a restore using CIFS, which led to errors, and when I looked at the file sizes of the corrupted files:
Whats interesting is that 3145728 bytes is exactly 3MB, and 2097152 bytes is exactly 2MB. Something weird is going on that is truncating a handful of files to precise even megabyte sizes.
This issue isn’t limited to my NAS, apparently. I restored the same 4GB directory directly to the root file system on the server running Duplicati and as expected, it worked fine a second time, no errors, all files are an exact match to the “live” files on the backup source. So far so good.
Then I copied those files to my NAS using Linux “cp -a” to see if something in the network transfer between the server and the NAS was creating problems. After the files copied over I compared them to the live files, and they were again a perfect match. I did this test twice, and both times all the files copied directly from the server to the NAS perfectly.
Finally, I created a share on a Windows machine, mounted it on the server running Duplicati, and ran a restore operation to that share. 15 files had errors, and most were even-megabyte final file sizes, all truncated from the correct file sizes in the source.
My conclusion is that from my server, Duplicati isn’t able to reliably write to CIFS/SMB shares (whether they are Windows or Linux shares). I switched all my shares to NFS, and everything works perfectly, so I’m going with that for now.
I was having issues backing up when I was using SMB/CIFS (I had a prior thread about hash errors that you were helping me with). The solution was to switch to FTPS for “uploading” to my NAS. This fits the overall issue I’m seeing: Duplicati can’t reliably write to SMB shares on my LAN, though it seems it can read from them. Somehow when writing, files are getting randomly truncated, whether they are dblock files during a backup or regular user files during a restore.
Oh, right - knew that. Sorry, it’s been along day for me.
So my current working SFTP backups are going to a local unRAID server with what it calls SMB shares. Technically, those are CIFS - right? If so, I should be able to try a restore from the SMB share (instead of SFTP) and see if I also get errors.
Does that make sense to you or am I comparing apples to oranges?
Apologies for my quick edits - after I posted I re-read and corrected a few things.
When you say SFTP backups going to SMB shares, what part of the backup is SFTP?
If I start from what I had before (my earlier thread):
Backup Operation:
Windows SMB share mounted via “mount -t cifs” on the Duplicati server (read-only): OK reading
Duplicati write backup files (dblocks, etc.) to ReadyNAS CIFS/SMB share: Failed with hash errors
but
Duplicati write backup files (dblocks etc) to ReadyNAS FTPS share: OK writing
Today I think I figured out:
Restore Operation:
Duplicati read from local USB drive, write restored files to CIFS/SMB share on ReadyNAS: failed with corrupted files
Duplicati read from local USB drive, write restored files to CIFS/SMB share on Windows 10 PC: failed with corrupted files
but
Duplicati read from local USB drive, write restored files to NFS share on ReadyNAS: OK writing
So for me, it’s not restoring from an SMB share, it’s writing to an SMB share that breaks, whether that’s writing dblock files or writing restored user files.
I’m probably not explaining it well; with the DST change I’m pretty tired myself!
I have Duplicati running on Windows backing up to a local SFTP destination running on Linux based unRAID. Aside from an issue not related to this, I haven’t had any issues.
I apologize in advance if I’ve totally got this wrong, but are you running Duplicati on the ReadyNAS and backing up your Windows contents via a share mounted on the ReadyNAS rather than running Duplicati on the Windows box itself?
Close. I am running Duplicati on an Ubuntu server.
Duplicati is backing up 3 windows machines, 6 shares on a NAS, and (so far) one remote Linux system. The Windows machines are mounted via SMB, the NAS is (now) mounted via NFS, and the remote Linux system is mounted with SSHFS.
Each backup goes to four destinations: Dropbox, B2 Cloud, and two removable USB drives.
Got it - so you’re running Duplicati on one “central” box and backing up everything else via “local” (to the Duplicati installed box) mounts. And I assume you’ve got (at least) 4 jobs - one for each of your destinations (Dropbox, B2, USB 1, USB 2).
So in your tests Duplicati was writing from the Linux box over a CIFS/SMB mount point to a Windows box when it had the failures.
If I do my testing, I would be writing from a Windows box to an SMB share - so not quite the same thing, but maybe worth a test anyway.
Exactly. Your ability to distill my situation down to a couple of clear sentences is heartening.
And yes, I have 3 to 4 jobs for each backup (B2 is new to me, it’s very affordable). I keep track of them all in an Excel spreadsheet so I can sort by start time, source, or destination.
The most interesting thing I learned yesterday was that it wasn’t just writing to the SMB shares on the NAS that was a problem; I also had data corruption writing to a Windows share that I created just for the test. And when I copied directly from the Ubuntu server to an SMB share there was no corruption, which may rule out general network stack/routing/switching, and points me in the direction of my Duplicati installation or Duplicati itself being the source of the problem. But assuming SMB shares are included in “File storage” as a backend in the stats, some of the 18737 reported backups yesterday must be to SMB shares so I’m back to only myself and one other user (from my other thread) seeing issues with SMB as a destination.
My testing method was to restore a ~4GB directory, containing various subdirectories and about 2000 files ranging from <1kb to 500MB, to a blank folder in a new destination, then run WinMerge to compare my original/live files against the restored set. I used WinMerge because Duplicati didn’t always throw an error for every corrupted file in the recovery folder so I wanted to capture all the differences. Most of them were corrupted JPG and PSD files that were only partially written, so I could see the top part of the JPG files for example.
Anywhere from three to ten files in each failed test (it varied). I just checked, it’s about 1600 files total. While that may be a small percentage, in general I’d consider any failure highly significant.
I agree - I’m just not sure I’ve got 4G of test data floating around so wanted to make sure I picked a source size that should reflect at least a few issues.
Edit:
I’ve got a 2.5G backup running on my Linux box using an SMB mounted Windows share as the source. Once that’s done I’ll try restoring back over the SMB mounted Windows share (to a different folder) and binary compare the two locations.
In case I haven’t already said it…just to be clear, it sounds like in all your testing it seems the backups (even with an SMB/CIFS mounted source) are working just fine - it’s the restore over the SMB/CIFS mount that has issues. Restoring from the same backup over something else (such as NFS) works just fine.
