Storing empty metadata

[Siri]

Hello together, I have the following problem:

I have a HDD connected to my FRITZ!Box 7530 via USB, from which I would like to make a backup via Duplicati (latest version) to a local folder on my Windows 10 PC. The HDD is mounted as drive “Y” under Windows via SMB (min. 2). When starting the backup I get the following error message:

2022-02-13 16:42:06 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for “Y:\Volume\Template Backup\New Text Document.txt”, storing empty metadata.

It copies all files, that fits so far.

I can only solve this problem if I set the option --skip-metadata=false, but then I lose the metadata when I restore the file (it will be reset to the date when I get the file from the backup). The problem only exists when I fetch files from this drive, 2 local test folders on the PC can be processed without errors, ergo I suspect that it must somehow have to do with the FRITZ!NAS.

Unfortunately, I have no idea what it could be, so I turn directly to you.

[ts678]

Welcome to the forum @Siri

@Siri

I suspect that it must somehow have to do with the FRITZ!NAS.

This might be relevant, and I wonder if @jfcb ever got with AVM?

@jfcb

I have the same problem as @x86 with the NAS on a fritzbox 7560. Fritzbox 7560 is a box with USB ports to attach hard drives, that I can then mount in Win 7.
The problem started when the fritzbox updated its OS to FRITZ!OS 07.27

@jfcb

AVM GmbH is not very reactive… I’ll try them if everything else fails.

duplicati/duplicati/blob/27ff8cd20b3e349f6efd7b3b754637c0d940ea9c/Duplicati/Library/Main/Operation/Backup/MetadataGenerator.cs#L79-L83

The one-line summary omits the exception details. Try watching About → Show log → Live → Warning.

If your Duplicati is running as a Windows service, as opposed to you in usual login session, let us know.

[jfcb]

@ts678

This might be relevant, and I wonder if @jfcb ever got with AVM?

No I did not contact AVM since I have an acceptable work around with --skip-metadata to make the warnings disappear, and I wouldn´t know exactly what question to ask AVM.

I did more tests with “activate support for SMBv1” “Unterstützung für SMBv1 aktivieren” in the Fritz!Box

I get the same number of warnings with and without activating support for SMBv1, see below Duplicati output, identical in both cases. These are the 20 warnings recorded in the journal:

    "2022-02-19 12:18:16 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\\", storing empty metadata",
    "2022-02-19 12:18:16 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\2019.05.13 CT JF oreille.iso\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\2019.05.13 CT JF oreille.mds\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\2019.05.13 traduction.rtf\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\2019.05.28 JF oreille.pdf\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\lisezmoi.txt\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\pour_benutzer_konto_JF.bmp\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\Thumbs.db\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\wlan_attacks.xls\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Aix crypté.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Aix fixe.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Aix portable.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\asus.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Desktop.ini\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Gabriel.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\gwiazda-Dell.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\gwiazdaMedion.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Michal.vnc\", storing empty metadata",
    "2022-02-19 12:18:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for \"Z:\\VNC\\Ordifixe-Medion.vnc\", storing empty metadata"

but there are 1459 warnings in total. Also, there are exactly 1458 files and directories on the network share.

In conclusion the SMB version set in the fritz!box appears irrelevant to the “storing empty metadata” message. I can live with no metadata as long as I can retrieve the data itself.

[ts678]

@Siri

2022-02-13 16:42:06 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.MetadataGenerator.Metadata-MetadataProcessFailed]: Failed to process metadata for “Y:\Volume\Template Backup\New Text Document.txt”, storing empty metadata.

One thing about Duplicati is that I think it asks for a very full view of the metadata, including permissions.

@drwtsn32

SMB will translate as best it can Linux permissions to something that Windows understands. It’s not a perfect translation due to inherent differences between Windows and Linux.

A question is what sort of Windows tools show permissions? In the other topic, Explorer dropped the tab.

@jfcb

Please see the snapshot of the Properties windows of a NAS file: there is no Security tab.

and looking at a FAT32 USB stick I plugged in, it does the same, yet Duplicati asks for permissions. It got

{“win-ext:accessrules”:“[{"Rights":-1,"ControlType":0,"SID":"S-1-1-0","Inherited":false,"Inheritance":0,"Propagation":0}]”,“CoreAttributes”:“Hidden, Archive”,“CoreLastWritetime”:“635854428920000000”,“CoreCreatetime”:“635854428811400000”}

That’s the exact content of file metadata dblock which in my case (unlike others’) didn’t error and bail out.
PowerShell is written in .NET Framework (I think) just like Duplicati, and is more capable than DOS tools.

Simple test in a PowerShell window (I’m not good at PowerShell, so anybody who is can feel free to join):

NTFS:

PS C:\backup source> $a = Get-Item A.txt
PS C:\backup source> $a.GetAccessControl()

Path Owner           Access
---- -----           ------
     HP4\Maintenance BUILTIN\Administrators Allow  FullControl...


PS C:\backup source> $a.GetAccessControl().Access


FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : BUILTIN\Administrators
IsInherited       : True
InheritanceFlags  : None
PropagationFlags  : None

FileSystemRights  : FullControl
AccessControlType : Allow
IdentityReference : NT AUTHORITY\SYSTEM
IsInherited       : True
InheritanceFlags  : None
PropagationFlags  : None

FileSystemRights  : ReadAndExecute, Synchronize
AccessControlType : Allow
IdentityReference : BUILTIN\Users
IsInherited       : True
InheritanceFlags  : None
PropagationFlags  : None

FileSystemRights  : Modify, Synchronize
AccessControlType : Allow
IdentityReference : NT AUTHORITY\Authenticated Users
IsInherited       : True
InheritanceFlags  : None
PropagationFlags  : None

FAT32:

PS E:\> $a = Get-Item mapfile
PS E:\> $a.GetAccessControl()

Path Owner    Access
---- -----    ------
     Everyone Everyone Allow  -1


PS E:\> $a.GetAccessControl().Access


FileSystemRights  : -1
AccessControlType : Allow
IdentityReference : Everyone
IsInherited       : False
InheritanceFlags  : None
PropagationFlags  : None

I guess you can try something like this on the SMB share to see whether something blows up during test.