Exclamation Mark on All Source Folders

patg84 · May 9, 2023, 12:45pm

Not sure what’s going on. Backups were working fine and fully automated until a few days ago, the Duplicati throws an error stating, “Unable to read data from the transport connection: The connection was closed. —> System.IO.IOException: Unable to read data from the transport connection: The connection was closed.”.

Now every folder under the C:\ (system drive for Windows) or D:\ (misc drives with random files/folders) drive (both SSD hdds) is coming up as a yellow triangle when I try to pick the source folder. Duplicati is running as a service on Windows Server 2012 R2.

I stopped the service, uninstalled Duplicati, removed the database files & folders, etc and started with a fresh service install. Same issue. Any ideas as to what causes this yellow triangle on the folders? I’ve read permissions but Duplicati is running as the SYSTEM user so it should have full access to those files under the C:\ & D:\ drives without an issue.

gpatel-fr · May 9, 2023, 2:42pm

Hello

do you remember that you posted about this very same problem 3 months ago ? From reading the archive you had a conflict with another backup software.
The warning on the folders means that the Web server can’t read the folder content. Maybe it’s because of snapshots (did you setup your Duplicati backup to initiate a snapshot before starting ?), or Duplicati don’t really run as a service, or something has messed up the access rights. A way to see what happens is to export the job as a command line, open a terminal window under c:\program files\duplicati2 (as administrator), and paste the command line, it can allow to get all messages as they happen.

ClarksonPowell · May 10, 2023, 10:16am

File and folder permissions: Make sure that the user who is running Duplicati (in this case SYSTEM) has sufficient permissions to access folders on C:\ and D:. Check the permissions and make sure the SYSTEM user has read and write permissions to these folders.
Antivirus software: Some antivirus software may block access to certain folders or files, which can cause problems when backing up. Check your antivirus software settings and make sure that Duplicati is allowed to work with folders on C:\ and D: drives.
Network access: Make sure the Duplicati service is allowed access to the network, as it may use network resources for backups or upgrades. Check the firewall and network connection settings on the server to make sure Duplicati has the necessary permissions to access the network.
Disk or file system errors: Disk or file system problems can cause errors when reading or writing data. Check disks for errors and correct them if possible. It is also recommended that you check the integrity of the file system and correct any problems found.
Update Duplicati: Make sure you have the latest version of Duplicati installed. Some known problems may be fixed in the latest updates.

patg84 · May 22, 2023, 1:28am

Ok so it’s got nothing to do with backup files created from Cobian. I can verify I’ve uninstalled, cleared out all files, etc for Duplicati on this OS. I’ve rerun the script I use to install Duplicati as a service on all new machines I’m running Duplicati on. To be clear, this is the only machine I’m experiencing issues on. I have Duplicati running on 50+ machines without an issue using this script. It’s the same as manually moving the files to make Duplicati run as a service.

It’s running .NET 4.8.03761 (verified through, “Determine which .NET Framework versions are installed - .NET Framework | Microsoft Learn”).

I’ve spun up a test VM and mimicked this exact setup and I cannot duplicate it.

I get the yellow triangle on any folder I want to backup. Here’s how this occurs: Duplicati is running as a service. I create the backup job to Wasabi and pick the folder I want. For example "C:\Files\ ". Everything looks good, I run the back and the files upload to Wasabi. If I go back and edit the job and look at the source folder it now has the yellow triangle on it. I can dump new files into this folder and run the backup and it updates the files on Wasabi’s end. Now at some point in the future this is going to randomly fail with the error above, “System.IO.IOException: Unable to read data from the transport connection”.

The yellow triangle will appear on any folder on either the primary or secondary SSD.

I’ve double checked permissions on the folder and the drive to the working VM and everything is identical. SYSTEM has access to all of these folders/files. I’ve checked effective permissions and everything looks good.

I’m at a loss and I need this to work. Any help would be appreciated.

gpatel-fr · May 22, 2023, 6:31am

well, this looks very much like a hardware problem.
I know that some software problems emulate very well hardware problems, but often it’s just a hardware problem. It may happen on 2 SSD, but SSD are not existing just by themselves, they are accessed through a (single) controller. IIRC this is a Win 2012 server ? the computer is probably quite ancient, maybe it is showing some age problem.

Did you check the system journal (filter on errors and warning for example) ? A test software may not find anything if it’s an intermittent problem (power troubles badly filtered by the computer for example, or external perturbation).

ts678 · May 22, 2023, 11:41am

Just to verify what you see, the below is a folder which allows Full control to SYSTEM and me.
SYSTEM needs explicit access (unlike root on Linux), and this view is from administrator Duplicati:

I zoomed the browser to get a larger image. Is this the exact triangle (on top of a folder) you have?

A way to witness an access problem is with Sysinternals Process Monitor, watching folder access.
While we can’t resolve all possible Windows or hardware issues, this may show what Duplicati got.

EDIT and in this case, the answer of no_administrators folder was a Windows ACCESS DENIED:

EDIT 2:

As an alternative to zooming icon, at least in Edge, so likely Chrome) one can right click → Inspect:

<a class="type x-tree-icon-locked" alt="Access denied" ng-class="{invisible: node.hidden, loading: node.loading}"></a>

Search for x-tree-icon-locked in C# code gets to:

github.com

duplicati/duplicati/blob/36c1261f4e8621ceaa7a3eadc15e39b6766b5707/Duplicati/Server/WebServer/RESTMethods/Filesystem.cs#L226-L244


      
          var accessible = isFile || canAccess(s);

          var isLeaf = isFile || !accessible || isEmptyFolder(s);

          

          var rawid = isFolder ? Util.AppendDirSeparator(s) : s;

          if (skipFiles && !isFolder)

              continue;

          

          if (!showHidden && isHidden)

              continue;

          

          tn = new Serializable.TreeNode()

          {

              id = rawid,

              text = SystemIO.IO_OS.PathGetFileName(s),

              hidden = isHidden,

              symlink = isSymlink,

              iconCls = isFolder ? (accessible ? (isSymlink ? "x-tree-icon-symlink" : "x-tree-icon-parent") : "x-tree-icon-locked") : "x-tree-icon-leaf",

              leaf = isLeaf

          };

then

github.com

duplicati/duplicati/blob/36c1261f4e8621ceaa7a3eadc15e39b6766b5707/Duplicati/Server/WebServer/RESTMethods/Filesystem.cs#L190-L209


      
          private static IEnumerable<Serializable.TreeNode> ListFolderAsNodes(string entrypath, bool skipFiles, bool showHidden)

          {

              //Helper function for finding out if a folder has sub elements

              Func<string, bool> hasSubElements = (p) => skipFiles ? Directory.EnumerateDirectories(p).Any() : Directory.EnumerateFileSystemEntries(p).Any();

          

              //Helper function for dealing with exceptions when accessing off-limits folders

              Func<string, bool> isEmptyFolder = (p) =>

              {

                  try { return !hasSubElements(p); }

                  catch { }

                  return true;

              };

          

              //Helper function for dealing with exceptions when accessing off-limits folders

              Func<string, bool> canAccess = (p) =>

              {

                  try { hasSubElements(p); return true; }

                  catch { }

                  return false;

              };

so it looks like something was found inaccessible.

ts678 · May 22, 2023, 2:21pm

Can’t access some folders, despite service running as system user
should dispel that standalone comment, although you did try further tests.
If you wish to (carefully) try a SYSTEM Command Prompt, recipe is there.

Also be aware that ACE ordering in an ACL might matter, as discussed at

How the System Uses ACLs

is an example of a test that I would hope would show an accurate answer.
It looks like Duplicati code above doesn’t look much at access error detail.

Process Monitor might give a better view of exactly what Windows rejects.
Note that the view may vary with altitude (where in the stack got watched):

Change Altitude of Process Monitor (ProcMon)

patg84 · June 7, 2023, 11:49pm

06/05/23: Here’s what has transpired since my last post.

Ran “accesschk64.exe” from sysinternals against the C:\ drive. Results show RW permissions on the folder(s) in question as the SYSTEM user. I then ran this against the subfolders in question and again I got the same RW result.

Process monitor does not show access denied in any scenario that I could try.

This is a fresh install of the operating system following a hard drive failure. I have replaced it with a Samsung EVO SSD. Haven’t had an issue with anything else on this machine so I highly doubt hardware or the main board is the culprit. Also FWIW the D:\ drive is the original enterprise Seagate traditional HDD (7200 RPM) and this drive has permission issues in Duplicati as well. Yellow triangle.

" Can’t access some folders, despite service running as system user" This guy is making files with admin permissions only and not the system user so that really doesn’t apply. My files have the system user as full access. Verified by Windows effective access.

Here’s what I can say for sure. I have recreated the Windows environment in a VM outside this server and duplicated the entire setup. I cannot get the VM to fail like on the actual machine when using Duplicati. Unfortunately I cannot just walk in and wipe this server as much as I would like to. It’s running several Virtual Machines under Hyper-V 24/7.

Initial setup of the job in Duplicati shows no yellow triangle. Do a dry run and it shows no errors but if you go back in and edit the job, now you can see the yellow triangle on any folder.

**Here’s something interesting. Upgraded from 2.0.6.3 (2.0.6.3_beta_2021-06-17) to 2.0.7.1 (2.0.7.1_beta_2023-05-25) and now backups to the S3 backend are completing and not hanging up on this machine. However the folders still show a yellow triangle under on the folders.

EDIT: On 06/07/23 the yellow triangles are gone when I go back in and look at the source folders. No clue what’s going on but it appears to be working now. Backing up Hyper-V machines at the moment. Time will tell.