How to deal with shadow files at NFS target?

Support
1

Hi there,

I am using duplicati on a Ubuntu VM to backup application data to a windows server NFS location.
Everything went well for some time but now I get an error message like the following:

System.UnauthorizedAccessException: Access to the path '/media/<redacted>/duplicati-nfs/.@GMT-2021.04.09-10:00:21' is denied. ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
  at System.IO.FileStatus.EnsureStatInitialized (System.ReadOnlySpan`1[T] path, System.Boolean continueOnError) [0x00039] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0 
  at System.IO.FileStatus.GetAttributes (System.ReadOnlySpan`1[T] path, System.ReadOnlySpan`1[T] fileName) [0x00000] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0 
  at System.IO.FileSystemInfo.get_Attributes () [0x0001c] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0 
  at System.IO.FileInfo.get_Length () [0x00000] in <9f0df102fe6e4cfea29d2e46f585d8a5>:0 
  at (wrapper remoting-invoke-with-check) System.IO.FileInfo.get_Length()
  at Duplicati.Library.Common.IO.SystemIOLinux.FileEntry (System.String path) [0x0000d] in <d9b5a2b54a2d4811a9f1d6efb66f507e>:0 
  at Duplicati.Library.Backend.File+<List>d__20.MoveNext () [0x00088] in <0ecf14088dc1405b8a31c2503a04de16>:0 
  at Duplicati.Library.Interface.BackendExtensions.TestList (Duplicati.Library.Interface.IBackend backend) [0x00017] in <fcf6c74b3a9546efbbb1e9dfb993a4e1>:0 
  at Duplicati.Library.Backend.File.Test () [0x00000] in <0ecf14088dc1405b8a31c2503a04de16>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.TestConnection (System.String url, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x000b7] in <c5f097a49c0a4f1fb0f93cf3f5f218b1>:0 
  at Duplicati.Server.WebServer.RESTMethods.RemoteOperation.POST (System.String key, Duplicati.Server.WebServer.RESTMethods.RequestInfo info) [0x00094] in <c5f097a49c0a4f1fb0f93cf3f5f218b1>:0 
  at Duplicati.Server.WebServer.RESTHandler.DoProcess (Duplicati.Server.WebServer.RESTMethods.RequestInfo info, System.String method, System.String module, System.String key) [0x00289] in <c5f097a49c0a4f1fb0f93cf3f5f218b1>:0

As I found out the “.@GMT-2021.04.09-10:00:21” file is a shadow copy file generated by windows volume shadow service (VSS).
If I list the mounted folder on the Ubuntu VM I can see all the duplicati files but it seems to have its problems with the shadow files:

2021-05-17 21_28_21-Window
2021-05-17 21_28_21-Window931×218 7.36 KB

2021-05-17 21_29_46-Window
2021-05-17 21_29_46-Window795×171 15.5 KB

VSS is generally needed on the windows server instance and there seem to be some drawbacks like performance issues when disabling it for specific folders.

Do you have a suggestion on how to deal with those shadow files?
Would the –no-backend-verification = true be of any help and if yes what would be the drawbacks? It would be important to me to know if the backup files on the backend were corrupt.

Thanks in advance if you can help me with this :slight_smile:

3

Interesting… I didn’t know Windows Server acting as an NFS server exposes shadow copies to Linux clients this way. There are no filtering options I’m aware of when it comes to the destination directory.

Is there any way to configure Windows to not expose these shadow copies to NFS clients? Other than that, disabling shadow copy creation on those folders seems like the most straight forward option. I don’t know about the performance penalty though.

The --no-backend-verification option may work, but it would increase the risk of broken backups.

Could you possibly use a protocol besides NFS? Maybe SMB?

4

Can you make a folder dedicated to the backup, and keep clutter out?
This is typically good practice, whether or not there’s an odd situation.

Alternatively I think shadow storage can be put on some chosen drive:

C:\>vssadmin resize shadowstorage /?
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2013 Microsoft Corp.

Resize ShadowStorage /For=ForVolumeSpec /On=OnVolumeSpec /MaxSize=MaxSizeSpec
    - Resizes the maximum size for a shadow copy storage association between
    ForVolumeSpec and OnVolumeSpec.  Resizing the storage association may cause shadow
    copies to disappear.  As certain shadow copies are deleted, the shadow copy storage
    space will then shrink.  If MaxSizeSpec is set to the value UNBOUNDED, the shadow copy
    storage space will be unlimited.  MaxSizeSpec can be specified in bytes or as a
    percentage of the ForVolumeSpec storage volume.  For byte level specification,
    MaxSizeSpec must be 320MB or greater and accepts the following suffixes: KB, MB, GB, TB,
    PB and EB.  Also, B, K, M, G, T, P, and E are acceptable suffixes.  To specify MaxSizeSpec
    as percentage, use the % character as the suffix to the numeric value.  If a suffix is not
    supplied, MaxSizeSpec is in bytes.

    Example Usage:  vssadmin Resize ShadowStorage /For=C: /On=D: /MaxSize=900MB
                    vssadmin Resize ShadowStorage /For=C: /On=D: /MaxSize=UNBOUNDED
                    vssadmin Resize ShadowStorage /For=C: /On=C: /MaxSize=20%

C:\>

If I recall correctly (you can search), Windows Server has some GUI tools for shadows that others lack.
ShadowCopyView is a third-party tool that can run on the server to look at its shadow copies with a GUI.
NFS Shadow Copies is (I think) a copy of an old article I’ve seen elsewhere, that might help understand.

In screenshot, the blue lines might be by NFS root squash to -2 (-2 16-bit=nobody, 32-bit=4294967294).
For the red lines, I wonder if the shadow copy has disappeared out from under NFS but it hasn’t noticed.