File System Warnings from backup jobs

What’s New in PowerShell 7.4 says it’s built on .NET 8.0.0. Duplicati 2.0.9.x is on some .NET 8. Windows 10 PowerShell (at least mine) is version 5.1 so .NET Framework like Duplicati 2.0.8.1.

If you can manage to get PowerShell 7.4 to fail on some test system, then maybe you can open https://github.com/PowerShell/PowerShell/issues to get Microsoft people working on solutions…

PowerShell devs probably know better how to work with .NET devs since it’s all inside Microsoft. There might still be some issues with the Netware code, but at least they should know Windows.

.NET problems (especially for current versions – not certain about Framework) can possibly see:

https://github.com/dotnet/runtime

but they might be more receptive if an issue came through PowerShell team, or other direct user.

While Duplicati is a user of the runtime, I don’t think we can isolate the issue as well as desirable, whereas you as a PowerShell end user can possibly do that, although it might take a lot of doing.

The Netware driver people probably know drivers, so they’re an option. Talk with the right people.

From a Duplicati point of view, this virtual drive is not functioning similarly enough to a local drive. There is no Duplicati logging at this level, so Process Monitor’s may be the best look that there is.

Where are you getting these ideas from? They might be right, but I’d like the citation or details .

Are extended attributes important to backup, or are you just aiming for a warning-free backup?

If you need to backup something that Duplicati can’t get (even without warning), maybe no luck.

Have you been able to verify on a warning-free file that a restore sets things up as you’re after?

I’ve been trying to get you to talk to (or read from) people who know about Netware. Have you?

Didn’t you pay someone at some point for the code that’s misbehaving? Do they have support?

While a Process Monitor finding is interesting, is it relevant, e.g. does it correlate with warnings?

EDIT 1:

Last question of relevance is even more a question, as I get this on a good backup of a local file:

8:40:11.5768752 AM	Duplicati.GUI.TrayIcon.exe	26336	19464	QueryRemoteProtocolInformation	C:\backup source\sub1\length1.txt	INVALID PARAMETER	0.0000086	8:40:11.5768838 AM	

0	FLTMGR.SYS	FltDecodeParameters + 0x210b	0xfffff802359764cb	C:\Windows\System32\drivers\FLTMGR.SYS
1	FLTMGR.SYS	FltDecodeParameters + 0x1bba	0xfffff80235975f7a	C:\Windows\System32\drivers\FLTMGR.SYS
2	FLTMGR.SYS	FltDecodeParameters + 0xc61	0xfffff80235975021	C:\Windows\System32\drivers\FLTMGR.SYS
3	FLTMGR.SYS	FltDecodeParameters + 0x66b	0xfffff80235974a2b	C:\Windows\System32\drivers\FLTMGR.SYS
4	ntoskrnl.exe	IofCallDriver + 0x55	0xfffff80236c4ad55	C:\Windows\system32\ntoskrnl.exe
5	ntoskrnl.exe	IoBuildDeviceIoControlRequest + 0x450	0xfffff80236ced8f0	C:\Windows\system32\ntoskrnl.exe
6	ntoskrnl.exe	NtQueryInformationFile + 0x7de	0xfffff8023702aade	C:\Windows\system32\ntoskrnl.exe
7	ntoskrnl.exe	setjmpex + 0x8628	0xfffff80236e12308	C:\Windows\system32\ntoskrnl.exe
8	ntdll.dll	NtQueryInformationFile + 0x14	0x7ffb3f30d714	C:\Windows\SYSTEM32\ntdll.dll
9	KERNELBASE.dll	GetFileInformationByHandleEx + 0xc7	0x7ffb3cf71117	C:\Windows\System32\KERNELBASE.dll
10	ntmarta.dll	AccRewriteSetEntriesInAcl + 0x16b4	0x7ffb3bb45834	C:\Windows\SYSTEM32\ntmarta.dll
11	ntmarta.dll	GetNamedSecurityInfoW + 0x316	0x7ffb3bb43316	C:\Windows\SYSTEM32\ntmarta.dll
12	ntmarta.dll	GetNamedSecurityInfoW + 0x1bc	0x7ffb3bb431bc	C:\Windows\SYSTEM32\ntmarta.dll
13	ADVAPI32.dll	GetNamedSecurityInfoW + 0x38	0x7ffb3d8227f8	C:\Windows\System32\ADVAPI32.dll
14	mscorlib.ni.dll	mscorlib.ni.dll + 0xc5968e	0x7ffb2a16968e	C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\9707a28127c32bb87f1d241096ec2d9f\mscorlib.ni.dll
15	mscorlib.ni.dll	mscorlib.ni.dll + 0xe3f941	0x7ffb2a34f941	C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\9707a28127c32bb87f1d241096ec2d9f\mscorlib.ni.dll
16	mscorlib.ni.dll	mscorlib.ni.dll + 0xe3efbb	0x7ffb2a34efbb	C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\9707a28127c32bb87f1d241096ec2d9f\mscorlib.ni.dll
17	mscorlib.ni.dll	mscorlib.ni.dll + 0xe40604	0x7ffb2a350604	C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\9707a28127c32bb87f1d241096ec2d9f\mscorlib.ni.dll
18	mscorlib.ni.dll	mscorlib.ni.dll + 0xe41210	0x7ffb2a351210	C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\9707a28127c32bb87f1d241096ec2d9f\mscorlib.ni.dll
19	<unknown>	0x7ffaccfb2911	0x7ffaccfb2911	

and none of this is Duplicati code. Process Monitor only shows the low-level effects of our code.

EDIT 2:

The stored representation of the metadata for that test file, obtained from the destination dblock:

{"win-ext:accessrules":"[]","CoreAttributes":"Archive","CoreLastWritetime":"637937788237568611","CoreCreatetime":"637280032989835983"}

and I think accessrules came from .NET Framework by the below:

File.GetAccessControl Method

If you right-click → Properties → Security → Advanced on a typical local Windows NTFS file, the rules are typically just inherited from some other folder, so there’s nothing to specifically say here.

Windows time stamps and file attributes should be more obvious. If you expect other things here, such as Netware attributes that Duplicati and .NET Framework aren’t made to handle, good luck.

Thanks for your note.

We have been running netware since 1988 and have a reasonable knowledge of the server, the client and the rights surrounding files on it.

As for the netware server side, the server has generated NO warnings at all on any of the server volumes or system logs. That is where I checked first. I actually open the console and run monitor, which allows me to watch the connection as duplicati is running. I see the files open, be read and closed without issue. I took the duplicati log and looked at about 20 of the error files and I don’t see anything odd in the source files except maybe extended file permissions. Originally when we started testing this, I knew the netware was decompressing files that had been stored for a long time. That could take from 0-8 seconds depending on the file size but there were no warning messages on this side and as long as duplicati waited for a response and did not time out, all was good. I ran Procmon and it confirmed NO error from files erver side. The error is duplicati calling the remote storage server. The only issue I can presume is that extended file status or permissions do not fit the target system or are not stored properly?? Complete conjecture but I looked the server logs from the storage server and there are NO entries in the error log. Does Duplicati store status somewhere on the system? How can I log the errors in a deeper way?

The system running this is a windows Pro 10 22H2, build 19045.5011 Running a netware client Client for Open Enterprise Server 2 SP7 (IR3)

As always, I appreciate your help and am happy to look into any possibility to help out the duplicati project.

Thank you!

The only Netware queries in this forum or GitHub Issues are yours. I sure have not run it.
The rareness of Netware unfortunately brings increased difficulty in trying to provide help.

Thank you for providing further details on testing below.

The only files whose data would be read are ones whose timestamp or attributes changed.
I don’t know if attempting to get file metadata requires Netware to decompress the file first.
Possibly you can draw conclusions by seeing what files get read compared to file changes.

This part I was able to get from Google search, but I didn’t manage to find it doing timeouts.

Further, the lead author said above that no timeouts are managed by Duplicati. It just reads.
This is blaming a lot on a timeout, including extended attribute issue we’re discussing more.
Process Monitor should have a lot of time stamps. I can’t see the timing from a single query.

What would such an error look like? That output is hard to read.

Duplicati has no knowledge of or direct call to any remote storage server. You configured it as
Local folder or drive and it is treated as such, and expected to work. Yours is not working well.

Problem could be in .NET (as you said one time), Windows, or its installed driver for Netware. Tentatively I’ll say that your checking on the server seeing no errors suggests client-side error, somewhere below Duplicati where we have little to no ability to diagnose beyond what you did.

Too vague, but yes it stores status of many things in many places. It also offers (too many) logs detailing activity at the Duplicati level. It does not and likely cannot record what happens below, which is out of scope for an application program, and out of scope for Duplicati support as well.

A stack trace is sometimes possible and at least makes it easier to find the trail inside Duplicati. Detailed logs (but not beyond Duplicati) can be made with log-file=<path> and log-file-log-level probably raised some. If not used, Warning is the default. That might get a bit more information. Verbose will start talking about source files. The below is About → Show log → Live → Verbose which is reverse chronological. The log-file is chronological, has seconds, and runs unattended.

Oct 22, 2024 12:12 PM: File has changed C:\backup source\short.txt
Oct 22, 2024 12:12 PM: Checking file for changes C:\backup source\short.txt, new: False, timestamp changed: True, size changed: True, metadatachanged: True, 10/22/2024 4:12:04 PM vs 10/22/2024 4:09:34 PM
Oct 22, 2024 12:12 PM: Including source path: C:\backup source\short.txt

Profiling level gets extremely long and is aimed at SQL work. There’s an option to make it bigger.
I’d probably start with Verbose, as you care about file paths. Name may show up at different time locations because there is asynchronous processing at work doing the different pieces of backup.

If you make a huge log file, use your favorite huge log file viewer. I use glogg on some huge files.

Ok…so more details for the owner of the server company! The next item to check is to see if duplicati actually includes the file into the .zip file written to the storage server. How would I do this?

Please check if there is any debug mode option. When I check storage2200 system and the FTP logs

  • I see no entry for 30478_page242_245. There may be a connection but this file appears to not be uploaded.

There are connections and logs for files like
xferlog.legacy:Mon Oct 21 22:08:17 2024 0 72.93.86.242 4660177 /home/st30335/backup/jemjom/duplicati-b8e008f030bca4ecea5ea74f48e912c3c.dblock.zip b _ i r st30335 ftps 0 * c
xferlog.legacy:Mon Oct 21 22:08:19 2024 0 72.93.86.242 5852 /home/st30335/backup/jemjom/duplicati-ic73b0ec0adf34f1cb6b1648d206adcb8.dindex.zip b _ i r st30335 ftps 0 * c
xferlog.legacy:Mon Oct 21 22:08:20 2024 0 72.93.86.242 2286283 /home/st30335/backup/jemjom/duplicati-20241022T020001Z.dlist.zip b _ i r st30335 ftps 0 * c
xferlog.legacy:Mon Oct 21 22:08:44 2024 0 72.93.86.242 2286283 /home/st30335/backup/jemjom/duplicati-20241022T020001Z.dlist.zip b _ o r st30335 ftps 0 * c
xferlog.legacy:Mon Oct 21 22:08:44 2024 0 72.93.86.242 54945 /home/st30335/backup/jemjom/duplicati-i1043c1fb2f2f4a58ba13b4a693b12f1d.dindex.zip b _ o r st30335 ftps 0 * c
xferlog.legacy:Mon Oct 21 22:08:49 2024 4 72.93.86.242 52352543 /home/st30335/backup/jemjom/duplicati-b33bfd9ea5380484a9d5c2e38b7577817.dblock.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:22 2024 0 72.93.86.242 7526 /home/st30335/backup/jemjom/duplicati-b10eda27fa8624cc192a33e53d391ec73.dblock.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:24 2024 0 72.93.86.242 1263 /home/st30335/backup/jemjom/duplicati-i940706501013473c955911a1354b1f79.dindex.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:26 2024 0 72.93.86.242 2285717 /home/st30335/backup/jemjom/duplicati-20241015T020000Z.dlist.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:37 2024 0 72.93.86.242 2285717 /home/st30335/backup/jemjom/duplicati-20241015T020000Z.dlist.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:37 2024 0 72.93.86.242 93592 /home/st30335/backup/jemjom/duplicati-i7835314458ea4db59735ca9a59847817.dindex.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Mon Oct 14 22:07:43 2024 4 72.93.86.242 52418213 /home/st30335/backup/jemjom/duplicati-b7b8741c96222450ca1fdbfb2f3d17ffb.dblock.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:06 2024 0 72.93.86.242 4334920 /home/st30335/backup/jemjom/duplicati-bd0fc7d3a530741ed946541516ae5fd5d.dblock.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:08 2024 0 72.93.86.242 9972 /home/st30335/backup/jemjom/duplicati-ifcfc8393342d4c818eeb5e4464913049.dindex.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:09 2024 0 72.93.86.242 2286030 /home/st30335/backup/jemjom/duplicati-20241017T020000Z.dlist.zip b _ i r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:25 2024 0 72.93.86.242 2286030 /home/st30335/backup/jemjom/duplicati-20241017T020000Z.dlist.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:25 2024 0 72.93.86.242 53192 /home/st30335/backup/jemjom/duplicati-i5933b29a34364069b3932d6cd83fd999.dindex.zip b _ o r st30335 ftps 0 * c
xferlog.legacy.1:Wed Oct 16 22:08:31 2024 4 72.93.86.242 52379539 /home/st30335/backup/jemjom/duplicati-bfd5316ab3e1f45ce970af137182e063d.dblock.zip b _ o r st30335 ftps 0 * c

None are listed as ‘pdf’ in any logs.

John Quaglieri
InterServer, Inc

I don’t know what server company or owner you mean, but details are usually helpful here.

Restoring files from a backup and either use the search box or expand the folders to find it.
You don’t have to actually restore, just look. Harder method is Commandline and run a find.

If you really want to check the .zip for name, there’s a filelist.json file inside with names.
If you need to decrypt, you can run Duplicati’s SharpAESCrypt.exe CLI, or get GUI AES Crypt.

What is that and how does an FTP log relate to anything? The rest of this is meaningless to me.

I partially puzzled out what you wrote. From past topic in June, the ftp is Duplicati storage Destination, and part of the post here is correspondence with them (looking at posted sig).

I don’t know how /home/st30335/backup/jemjom gets in. Maybe folder path on storage?

who is listed on their web site as CTO.

You won’t see Duplicati uploading pdf files. All files should be of the form shown, with dblock and dindex as necessary for the volume of changes backed up, and a dlist at the end with file details.

Note that you’re not really supposed to have to peek inside the .zip files, but you asked about it. Source file changes are in default 100 KB (max) blocks in dblock files, and a dblock has a dindex.

The backup process explained (simplified)
How the backup process works (technical)

Database and destination internals (developer level)
Channel Pipeline (developer level but details what I meant when I said asynchronous processing)

If we can somehow reproduce the problem that would help tremendously with fixing it, but unfortunately that is not always the case.

I do have access to a number of different operating systems for testing, so if I can somehow replicate your situation on one of my setups, I can try to fix it.

It looks like NetWare is discontinued unfortunately. Is there an alternative that I can set up with some compression or similar tweaks to make it behave the same as yours?

Based on the error messages, and your description it looks like Duplicati attempts to read the non-extended attributes of the file, and this triggers some kind of error due to the file being not immediately available. Second run then has no problems as the file is “hot”.

I have tried to find similar issues but it is always something esoteric that is only linked to a single product.

I think this is case-by-case if you want extended file metadata or not. Restoring permissions is certainly a tricky operation.

I think the problem is more related to the network driver. It just happens that the attribute reading is done first, so that is where it is triggered. It could be related to extended attributes, but that could be tested by having some files with and some without. Seeing that this is a bit delayed, it might take a while before there is conclusive results.

Thanks. Netware was been replaced with Open Enterprise Server. The OES driver is currently and supported.

As mentioned before, there are no file read errors on the server. That said, it may be helpful to compare the error logs and determine if there are files that consistently cause the error and then look at NDS file attributes. One question that comes to mind is file locking. Is it possible duplicati is doing overlapped file access and running into a file lock issue? I do not believe that files are marked sharable by default. Thanks for your time and efforts.

In recent years, it seems this is based on SUSE Linux Enterprise Server, so supports CIFS/SMB.

CIFS Features and Capabilities (manual from Micro Focus, which is now acquired by OpenText)

I’ve been asking if it works, allowing backup without either sort of warning. Only you can test that.

EDIT:

Yes it’s a workaround, but as I’ve been saying, I’m not confident in being able to fix it in Duplicati.

I’d still like to hear of your trouble with other .NET applications. What problems are they noticing?

I mentioned PowerShell, and if you want to do something to see if an error or file warmup occurs,

Get-ChildItem with -Recurse might be a nice experiment, but likely does less than Duplicati use.