Duplicati can't access file even after access rights have been granted

Thereā€™s nothing for Duplicati to realize - it has exactly what access the share is allowing.

If this is the only file causing the error then I agree thereā€™s something goofy going on, but if Duplicati can get to even one file on the share then we know itā€™s not a share level permissions issue.

Is it possible somebody / something else on or using the NAS has the file open and itā€™s really a ā€œfile in useā€ issue? Itā€™s possible the share canā€™t adequately define that as a situation so it just tells Duplicati ā€œno, you canā€™t have the contents of this file but Iā€™m not going to tell you whyā€ā€¦

What version of Mono are you on? I just ran into a few weird file permission problems that seem to have gone away with an upgrade to 5.4.1.6-9 (I was on 3.1).

No, I donā€™t see how this is possible, since I can do any read operation I want with the file via windows explorer: I can open it and look at the contents, and most importantly, I can copy it to another folder.

Iā€™m not using mono. Iā€™m running duplicati on windows.

How do you have Duplicati accessing the NAS content - is it one of these?

  • mount point (mklink or symlink) as a subfolder
  • mapped drive letter
  • UNC path
  • Swish (or something similar)
  • voodoo magic

At this point, Iā€™m actually pretty sure itā€™s voodoo.

But seriously: this is actually a good point. Because duplicati is accessing it via UNC path:

While when I access it manually, it is via mapped drive. (The reason why duplicati is using the UNC path is because it just wonā€™t see the mapped drive)

However, I repeated the tests by accessing the file manually via the UNC path and it works just fine.

The fact that you donā€™t see the mapped drive in the Duplicati file picker, seems to be an indication that the Duplicati server/service is running in a different user context than the account you logged on with.
This could explain why you can access the file interactively in contrast to Duplicati.
Are you 100% sure that you granted read permissions to the account that is used to start the Duplicati server/service?

I agree with kees-z, but only if you canā€™t back up ANY files over this UNC connection.

If other things on the share are getting backed up EXCEPT this default.m3lib file then I thing something else is going on.

Just out of curiosity, are you running MusicIP on your NAS and, if so, does stopping that service allow the backup to run as expected?

Yes. The service is running under the same user name as I am logged in with, i.e. my windows username:

Thatā€™s the thing. The backup job has successfully backed up about 600 GB:

image

The absolutely only file that fails is that one.

Yes, Iā€™m running MusicIP on my NAS (well spotted!) but I donā€™t know for sure what happens when I stop the service, but I thought it just doesnā€™t make sense that it would change anything, since the file is obviously not locked. The thing is, I only fiddle with the MusicIP settings every other year or so and every time I have to look things up again, like how to start or stop the service etc. So I canā€™t check it right awayā€¦

Thatā€™s fine. At the moment I only have a theory that MusicIP has the file open so the share isnā€™t allowing Duplicati access to it.

I realize this doesnā€™t match up with you being able to open the file manually, but itā€™s possible Duplicati is trying to exclusively open the file (so nothing else changes it while Duplicati is processing it) while the manual test open is fine with sharing access to the file.

On a local Windows drive the shadow copy service (if enabled) would handle sharing access while ensuring Duplicatiā€™s view of it doesnā€™t change while open, but that doesnā€™t generally extend to network connections.

If itā€™s only the file default.m3lib youā€™re having problems with, I guess @JonMikelV is on the right track.
In case it still turns out to be a permission problem (I donā€™t expect that), check if the location on your NAS is stored in the Windows Credential Manager.
The current version of Duplicati doesnā€™t make use of the Credential Manager, but there are plans to use the CM for storing passwords in future versions. So if credentials for accessing your NAS are stored in the CM, Duplicati isnā€™t aware of it.

Aaah, I didnā€™t know that there are different kinds of file access (exclusive and non-exclusive). So file access is not just a matter of granting access but also about requesting (the right kind of) access. :exploding_head:

Iā€™ll check it.

Yes, files that are opened exclusively cannot be read by other processes until the file is closed.

This is where VSS and LVM come in. These technologies create a consistent snapshot of all files, including open files. This snapshot is then used as backup source, making it possible to backup open files.

Snapshots do not work on remote shared folders, so you have to terminate the process that keeps the file open, or exclude the file from your backup (which seems to be the best approach for your situation).

Yes, I am aware of this. But @JonMikelV added something else to that picture: a process can request either exclusive access (and be rejected because anothee process already has (non-) exclusive access) or it can request non-exclusive access (and receive it even though another process already has (non-) exclusive access).

As far as I know, Duplicati does not open files exclusively. Read-Only access is the only requirement, which is not enough to open the file exclusively.
Snapshot technologies (VSS, LVM) are used to be sure that files donā€™t change while they are backed up.
I guess itā€™s not Duplicati, but MusicIP that opens the file exclusively.

No, becauss then I wouldnā€™t be able to open it eitherā€¦

That makes sense. In that case, I have no idea what blocks access to that file.
Itā€™s still strange that you donā€™t see your mapped drives in the file pickerā€¦ Iā€™ve tried it with my setup, when I start Duplicati.Server.exe using my own user account, I can see my mapped network drives in the file picker when selecting source files. So somehow your user account seems to be different from the account used to start Duplicati on your computer.

Okay, this is embarassing: the solution was there in front of our eyes all along. :flushed: Look at this:

And then this:

https://forum.duplicati.com/uploads/default/original/1X/89e3caca9c2c141ba18afc869829394fffaddb13.png

and this:

image

Notice something?

My default.m3lib is in the media share. But the error came from the together share (apparently I had copied the file there some time ago as a temporary backup and never deleted it. I didnā€™t realize that and kept looking at the wrong file. Iā€™m sorry for stealing your time with this.

I can say though, that I did learn some stuff, so thanks for that.

So here is what happened when I adjusted the access rights for the file that was actually causing the errors in the first place:

  1. The backup job completed with a fatal error:
8 Dec 2017 21:42: Message
Fatal error
Duplicati.Library.Interface.UserInformationException: Found 1 files that are missing from the remote storage, please run repair
   at Duplicati.Library.Main.Operation.FilelistProcessor.VerifyRemoteList(BackendManager backend, Options options, LocalDatabase database, IBackendWriter log, String protectedfile)
   at Duplicati.Library.Main.Operation.BackupHandler.PostBackupVerification()
   at Duplicati.Library.Main.Operation.BackupHandler.Run(String[] sources, IFilter filter)
  1. I did not run repair but just ran the backup again. And this time it finished without any error. Phew! :sweat_smile:

However, it looks to me like the ā€œfatal errorā€ was caused by the file not being backed up due to lack of access permission. In other words:

(i) duplicati backs up files but fails to access one of them and issues message (itā€™s not even a warning, which is also strange).
(ii) when the backup runs again it finds that file missing from the backup archive, thinks the database must be corrupted and asks for a repair.
(iii) when the backup runs yet another time, the file is correctly backed up (due to changed access rights) and there are no more errors.

My question: instead of assuming that there is a problem with the database, shouldnā€™t duplicati have ā€œrememberedā€ that the file couldnā€™t be backed up?

My guess is that when a file fails to be backed up, duplicati nonetheless writes it into the database as part of the backup. So this is probably what needs to be changed: only write files into the database if upload succeeded. That might save us a lot of ā€œfatal errorsā€ and unnecessary db repairsā€¦

1 Like

It depends on how much data is retrieved. For most failures, Duplicati will just omit it.

I am a bit confused. The error message seems to be related to the backup files (the dblock, dindex and dlist files) but the rest of the post is about a source file?

Which error message are you talking about?

By ā€œsource fileā€ you mean the default.m3lib?

Reading through my last post, I realize that it may not been clear enough in distinguishing two parts f the post:

  1. saying that large parts of the preceding discussion possibly were unnecessary because I was looking at the wrong file and
  2. describing a new problem that I observed (while looking at the correct file).

The new problem (which can probably be understood without reading the previous discussion) is:

And my interpretation is:

I donā€™t know how this translates into dblock, dindex and dlist files, Iā€™m basically just saying that duplicati shouldnā€™t suggest a repair, when it runs just fine without that repair.

That is highly peculiar, because the verification code that runs is exactly the same. If you run it more than once, it should fail or succeed the same way all the time, unless the remote data has changed.

I agree that it should not suggest it.

This is the part that confused med, because (i) seems to be related to the source file not being accessible, which is unrelated (?) to the other problem.

I do not understand how the permissions can affect this. In the check, Duplicati just lists the files and does not care if it can read them or not. For the error to occur, the file listing must exclude the file from the list.