Any idea what would cause this? Trying to recover my backup environment when the NAS storing the backups went down, and I’m left with this problem when I try to run a repair and even a delete+repair. This time I have also tried renaming the original remote directory and creating an empty one, and creates new backup files. I’m using my root account on the machine to create the directories/folders on the remote NFS share so I know my access is ok, yet Duplicati won’t:
System.IO.IOException: Input/output error
at Duplicati.Library.Main.BackendManager.List () [0x00049] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.DoRun (Duplicati.Library.Main.Database.LocalDatabase dbparent, System.Boolean updating, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00084] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.RecreateDatabaseHandler.Run (System.String path, Duplicati.Library.Utility.IFilter filter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+NumberedFilterFilelistDelegate filelistfilter, Duplicati.Library.Main.Operation.RecreateDatabaseHandler+BlockVolumePostProcessor blockprocessor) [0x00037] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.RepairHandler.RunRepairLocal (Duplicati.Library.Utility.IFilter filter) [0x000ba] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.RepairHandler.Run (Duplicati.Library.Utility.IFilter filter) [0x00012] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass18_0.<Repair>b__0 (Duplicati.Library.Main.RepairResults result) [0x0001c] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0026f] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x00007] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller.Repair (Duplicati.Library.Utility.IFilter filter) [0x0001a] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Server.Runner.Run (Duplicati.Server.Runner+IRunnerData data, System.Boolean fromQueue) [0x003ad] in <6e9570d287354b91a47f22584b706965>:0
v2.0.5.110_canary_2020-08-10 running on Fedora Server 32
Root squash[2][3] is a special mapping of the remote superuser (root) identity when using identity authentication (local user is the same as remote user). Under root squash, a client’s uid 0 (root) is mapped to 65534 (nobody). It is primarily a feature of NFS
It’s not clear which things are being done as root on the client over NFS, versus root on the NAS (where typically root has full access, while root on the client maybe gets even more restricted than usual user).
Ok, so how do I test this? I don’t quite follow what you wrote.
I successfully completed a brand new backup of the same files/folder to a new directory on the NFS share - when it finished I ran a repair and it looked fine. It came to do the next scheduled backup and failed again with the input/output error. I’m completely lost here.
I may have found a fix - I granted the Everyone group on the Windows server Modify rights to the folder shared with NFS. So far so good and the NFS share is locked to the IP of the machine accessing it so I’m not that concerned about the workaround.
The idea was that manual access (e.g. ls) as client root over NFS to a server might also error if a server treats root UID 0 as weak instead of super powerful. If you want root to stay root, there’s typically a way to say so. Linux server information is plentiful (start at linked Wikipedia article). Windows server info is not…
The basic question is probably how to do user mapping from a Linux user to (maybe different) server user.
I’m glad you have a workaround to live with server behavior. That might be configurable but I can’t say how.
The annoying thing is that the two Raspberry OS Pi I also use Duplicati with have no such issues, just this Fedora server - though it doesn’t surprise me as I regularly coming across bizarre issues with it, so perhaps this might be the last iteration of Fedora I use and migrate to something else next time.
If they’re running the same way as root on those, then I’m not sure how they avoid server remap of root.
If they’re running as a non-root user, server mapping might differ, but they probably remap to something.
Failed: Input/output error
Details: System.IO.IOException: Input/output error
at Duplicati.Library.Main.BackendManager.List () [0x00049] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.RemoteListAnalysis (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x0000d] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList (Duplicati.Library.Main.BackendManager backend, Duplicati.Library.Main.Options options, Duplicati.Library.Main.Database.LocalDatabase database, Duplicati.Library.Main.IBackendWriter log, System.Collections.Generic.IEnumerable`1[T] protectedFiles) [0x00000] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.BackupHandler.PreBackupVerify (Duplicati.Library.Main.BackendManager backend, System.String protectedfile) [0x0011d] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Operation.BackupHandler.RunAsync (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x01042] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at CoCoL.ChannelExtensions.WaitForTaskOrThrow (System.Threading.Tasks.Task task) [0x00050] in <9a758ff4db6c48d6b3d4d0e5c2adf6d1>:0
at Duplicati.Library.Main.Operation.BackupHandler.Run (System.String[] sources, Duplicati.Library.Utility.IFilter filter, System.Threading.CancellationToken token) [0x00009] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller+<>c__DisplayClass14_0.<Backup>b__0 (Duplicati.Library.Main.BackupResults result) [0x0004b] in <9d37c106a6af4e2db9ebdc93583bad34>:0
at Duplicati.Library.Main.Controller.RunAction[T] (T result, System.String[]& paths, Duplicati.Library.Utility.IFilter& filter, System.Action`1[T] method) [0x0011c] in <9d37c106a6af4e2db9ebdc93583bad34>:0
Is there any way to use an SMB share from a Linux Duplicati install, I could not see it as an option in the job settings? Just fed up with all this NFS nonsense.
Ok, so I switched the NFS mount at boot to an SMB mount and so far so good - but I’ve been here before.
Interestingly though, while looking at the NFS mounts on the Fedora and Raspberry installs, I saw that Fedora was connecting as NFS3 (and actually throws an NFS4 error during boot), whereas the Raspberry is connecting as NFS4. Definitely need to rethink dumping Fedora but I have quite a lot running on that box that it’s not trivial.
Just point to the share that you mounted. Note though, that SMB can sometimes be unreliable.
Just as with NFS, there are lots of SMB versions and variations and tunings that may factor in.
I tried that but could not get any combination of destination format or credential format to work - I did read in the docs that in fact SMB is not supported as a destination, which is why a looked into and set up the SMB mount at boot.
Are these both talking about SMB? The latter says it was set up. The former says nothing worked.
SMB might have a similar user mapping challenge to NFS, assuming SMB is going into Windows.
I don’t have any Linux SMB, so all I can do is point to docs. Internet search would surely find more.
Basically, Duplicati has neither an SMB client nor an NFS client built in. It uses the OS mechanism.
If you can get that to where a user running as Duplicati’s user seems to work, Duplicati might also.
SMB, though, is sometimes less reliable than desired, especially in the face of intermittent access.
Looks like I might need to delve into finding out why NFS4 isn’t working because it seems SMB is giving some issues as well. Not sure why these two files in particular because there is a file dated between those two which it doesn’t mention:
LimitedWarnings: [
2020-08-28 07:05:27 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /srv/nfs/backups/huginn/huginn-backup-20200828_070007.sql,
2020-08-28 07:05:27 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/srv/nfs/backups/huginn/huginn-backup-20200827_070006.sql", storing empty metadata
]
LimitedErrors: []
Log data:
2020-08-28 07:05:27 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-PathProcessingError]: Failed to process path: /srv/nfs/backups/huginn/huginn-backup-20200828_070007.sql
Mono.Unix.UnixIOException: Interrupted system call [EINTR].
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <f01ab055820c478fbd2bfa649991ba29>:0
at Mono.Unix.UnixFileSystemInfo.GetFileSystemEntry (System.String path) [0x0000f] in <f01ab055820c478fbd2bfa649991ba29>:0
at UnixSupport.File.GetFileType (System.String path) [0x00001] in <3a2c307381104a73ba105ad3712d666f>:0
at Duplicati.Library.Snapshots.NoSnapshotLinux.IsBlockDevice (System.String localPath) [0x00006] in <594d27435071428ca8c4e9243bb18091>:0
at Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess.AttributeFilter (System.String path, System.IO.FileAttributes attributes, Duplicati.Library.Snapshots.ISnapshotService snapshot, Duplicati.Library.Utility.IFilter sourcefilter, Duplicati.Library.Main.Options+HardlinkStrategy hardlinkPolicy, Duplicati.Library.Main.Options+SymlinkStrategy symlinkPolicy, System.Collections.Generic.Dictionary`2[TKey,TValue] hardlinkmap, System.IO.FileAttributes fileAttributes, Duplicati.Library.Utility.IFilter enumeratefilter, System.String[] ignorenames, System.Collections.Generic.Queue`1[T] mixinqueue) [0x00000] in <9d37c106a6af4e2db9ebdc93583bad34>:0
2020-08-28 07:05:27 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for "/srv/nfs/backups/huginn/huginn-backup-20200827_070006.sql", storing empty metadata
Mono.Unix.UnixIOException: Interrupted system call [EINTR].
at Mono.Unix.UnixMarshal.ThrowExceptionForLastError () [0x00005] in <f01ab055820c478fbd2bfa649991ba29>:0
at Mono.Unix.UnixFileSystemInfo.GetFileSystemEntry (System.String path) [0x0000f] in <f01ab055820c478fbd2bfa649991ba29>:0
at UnixSupport.File.GetUserGroupAndPermissions (System.String path) [0x00001] in <3a2c307381104a73ba105ad3712d666f>:0
at Duplicati.Library.Common.IO.SystemIOLinux.GetMetadata (System.String file, System.Boolean isSymlink, System.Boolean followSymlink) [0x00068] in <529b5bd32bc049799610792c1b48b8d7>:0
at Duplicati.Library.Snapshots.NoSnapshotLinux.GetMetadata (System.String localPath, System.Boolean isSymlink, System.Boolean followSymlink) [0x00000] in <594d27435071428ca8c4e9243bb18091>:0
at Duplicati.Library.Main.Operation.Backup.MetadataGenerator.GenerateMetadata (System.String path, System.IO.FileAttributes attributes, Duplicati.Library.Main.Options options, Duplicati.Library.Snapshots.ISnapshotService snapshot) [0x0001b] in <9d37c106a6af4e2db9ebdc93583bad34>:0
Original issue was on the backup destination side using NFS, unable to list the files. We worked on this:
System.IO.IOException: Input/output error
at Duplicati.Library.Main.BackendManager.List () [0x00049] in <9d37c106a6af4e2db9ebdc93583bad34>:0
New issue is on the backup source side, which I don’t think I knew was also remote. This particular issue needs some chasing down by someone, possibly someone who can run strace to see what’s interrupted:
EDIT: skip-metadata=true was also mentioned, and is easier test. Just set the option true to see if it helps.
FileAccessError while processing files explains the need for investigation, and ideally a reliable test case.
You’re pretty busy now trying to get things running right, but if you can help investigate, it might help solve.
I’ve tried another test.
Restarted both machines, server (Windows) and client (Fedora).
Wiped the backups off the remote Windows server and recreated the folder plus the NFS share.
Deleted the database for the job and started it.
After a few hours I had a completed backup.
Waited for the next scheduled backup, it failed with Input/output error again.
Dismounted/remounted share on client.
Manually started backup.
Completed successfully.
Going to leave it like that and see if the next scheduled backup fails again. If it does, remount, backup and remount again before waiting.
Quick update. So the next day, yesterday, the scheduled backup worked which was kind of annoying as it doesn’t then explain why today it failed.
So I tried restarting just Duplicati and a re-run failed as well - so the issue is not Duplicati directly but something it does or doesn’t do is affecting the mount.
Seems that doing:
umount /srv/nfs
mount -a
solves the problem as after that the backup is fine.
Any ideas how I can further trouble shoot this? Currently the fstab mount is set as:
Which is identical to the same ones I use for my two Raspberry Pi installs. I’ve also tried sync which still fails and even worse makes the backup extremely slow.
You might do better searching or asking on the wider Internet, except I’m not sure how you’d set context.
I’m not 100% clear on what error you’re seeing even in this latest post. I/O error? Source or destination?
You could try using strace to see what’s going on underneath. If destination. Duplicati logs can do more.
EDIT:
For destination error, logging at Retry level is good. For source, Verbose. Duplicati won’t log every read.
Trimming the backup down to a small size will also help, and most levels of log will make debug easier.
About → Show log → Live is good for easy cases. If it takes awhile, using a log file will likely be easier.
Stack traces can add context too. For a complete backup failure, try About → Show log → Stored too.