Repair and Rebuild database fails: Unexpected difference in fileset

dejhost · January 30, 2019, 8:16pm

Hello again. Sorry for the break in communication - had some troubles with my backup server. And you know: No server, no posts…
This is the logfile from the first day back online. Duplicati mentioned 170 warnings. Snapshots were set to “required”. Also the “write to logfile” option was turned on, but I receved an error: “logfile not accessable”. Will look into it.

Jan 30, 2019 8:29 PM: Result
DeletedFiles: 0
DeletedFolders: 0
ModifiedFiles: 6
ExaminedFiles: 53036
OpenedFiles: 18
AddedFiles: 12
SizeOfModifiedFiles: 3662924
SizeOfAddedFiles: 66524926
SizeOfExaminedFiles: 115256473687
SizeOfOpenedFiles: 70187850
NotProcessedFiles: 0
AddedFolders: 2
TooLargeFiles: 0
FilesWithError: 0
ModifiedFolders: 0
ModifiedSymlinks: 0
AddedSymlinks: 0
DeletedSymlinks: 0
PartialBackup: False
Dryrun: False
MainOperation: Backup
CompactResults:
    DeletedFileCount: 0
    DownloadedFileCount: 0
    UploadedFileCount: 0
    DeletedFileSize: 0
    DownloadedFileSize: 0
    UploadedFileSize: 0
    Dryrun: False
    MainOperation: Compact
    ParsedResult: Warning
    Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
    EndTime: 1/30/2019 8:27:07 PM (1548876427)
    BeginTime: 1/30/2019 8:26:14 PM (1548876374)
    Duration: 00:00:52.6502662
    Messages: [
        2019-01-30 20:23:04 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
        2019-01-30 20:24:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
        2019-01-30 20:24:38 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (5,42 KB),
        2019-01-30 20:25:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
        2019-01-30 20:25:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
...
    ]
    Warnings: [
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\QAQC - Documents\,
...
    ]
    Errors: []
    BackendStatistics:
        RemoteCalls: 8
        BytesUploaded: 5668474
        BytesDownloaded: 57871908
        FilesUploaded: 3
        FilesDownloaded: 3
        FilesDeleted: 0
        FoldersCreated: 0
        RetryAttempts: 0
        UnknownFileSize: 0
        UnknownFileCount: 0
        KnownFileCount: 5553
        KnownFileSize: 141034681193
        LastBackupDate: 1/30/2019 8:23:04 PM (1548876184)
        BackupListCount: 60
        TotalQuotaSpace: 0
        FreeQuotaSpace: 0
        AssignedQuotaSpace: -1
        ReportedQuotaError: False
        ReportedQuotaWarning: False
        ParsedResult: Warning
        Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
        Messages: [
            2019-01-30 20:23:04 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
            2019-01-30 20:24:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
            2019-01-30 20:24:38 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (5,42 KB),
            2019-01-30 20:25:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
            2019-01-30 20:25:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
...
        ]
        Warnings: [
            2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
            2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
            2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
            2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
            2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\QAQC - Documents\,
...
        ]
        Errors: []
DeleteResults:
    DeletedSets: []
    Dryrun: False
    MainOperation: Delete
    ParsedResult: Warning
    Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
    EndTime: 1/30/2019 8:27:07 PM (1548876427)
    BeginTime: 1/30/2019 8:26:03 PM (1548876363)
    Duration: 00:01:03.8985719
    Messages: [
        2019-01-30 20:23:04 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
        2019-01-30 20:24:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
        2019-01-30 20:24:38 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (5,42 KB),
        2019-01-30 20:25:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
        2019-01-30 20:25:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
...
    ]
    Warnings: [
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\QAQC - Documents\,
...
    ]
    Errors: []
RepairResults: null
TestResults:
    MainOperation: Test
    Verifications: [
        Key: duplicati-20190130T192304Z.dlist.zip.gpg
        Value: [],
        Key: duplicati-i09299b4398174a698aa76a4037e22cd4.dindex.zip.gpg
        Value: [],
        Key: duplicati-b6d999066c80e4f508459b2dc1ea8471a.dblock.zip.gpg
        Value: []
    ]
    ParsedResult: Warning
    Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
    EndTime: 1/30/2019 8:29:18 PM (1548876558)
    BeginTime: 1/30/2019 8:27:11 PM (1548876431)
    Duration: 00:02:06.4088496
    Messages: [
        2019-01-30 20:23:04 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
        2019-01-30 20:24:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
        2019-01-30 20:24:38 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (5,42 KB),
        2019-01-30 20:25:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
        2019-01-30 20:25:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
...
    ]
    Warnings: [
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
        2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\QAQC - Documents\,
...
    ]
    Errors: []
ParsedResult: Warning
Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)
EndTime: 1/30/2019 8:29:18 PM (1548876558)
BeginTime: 1/30/2019 8:23:04 PM (1548876184)
Duration: 00:06:13.6303117
Messages: [
    2019-01-30 20:23:04 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started,
    2019-01-30 20:24:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),
    2019-01-30 20:24:38 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (5,42 KB),
    2019-01-30 20:25:40 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
    2019-01-30 20:25:42 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b98fb25388e184e1a84f644f5d886233a.dblock.zip.gpg (313,94 KB),
...
]
Warnings: [
    2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
    2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\Templates - Documents\,
    2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
    2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\RD - Documents\,
    2019-01-30 20:23:43 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint\QAQC - Documents\,
...
]
Errors: []

Strange that this Sharepoint keeps popping up. Sharepoint synchronisation is enabled, but I haven’t moved any files for the past months.
Btw.: I use duplicati on the ubuntu computer of a family member. It uses the same transfer protocol (SFTP) to the same server. During a remote desktop session to the computer, I saw the following error popping up:

Unexpected result count: 0, expected 1

Seems familiar!? Maybe it has something to do with the protocol or the backup server? So I took a look at the servers logfile today, and I see plenty of these:

Jan 30 17:51:48 Jonas mountd[1935]: mount request from 192.168.1.14 for non existent path /Bicon
Jan 30 17:51:48 Jonas mountd[1935]: mount request denied from 192.168.1.14 for /Bicon
Jan 30 20:32:35 Jonas mountd[1935]: mount request from 192.168.1.14 for non existent path /XXX
Jan 30 20:32:35 Jonas mountd[1935]: mount request denied from 192.168.1.14 for /XXX

I am pretty sure it has something to do with duplicati, since the message stopped when duplicati was done doing backups. “Bicon” is a folder that I have on my computer - but I do not know why it should be causing problems.

dejhost · February 14, 2019, 1:36pm

Hello all,
Could somebody please help me to get this sorted out?
By now, I had to run a repair “database” again to get things working. It’s running now, but still shows 170 Warnings…

ts678 · February 14, 2019, 3:24pm

Did this happen? Last list of errors posted were on C:\Users\XXX\Documents\YYY\Sharepoint\SharePoint without the error details the logfile would have. About → Show log → Live during the backup might catch them if you can’t get --log-file working. Assuming you want these documents backed up, we need to know what the reason is that you can’t access them. You run Duplicati as UserName : YYY so FileAccessError could happen when trying to access XXX files (if that’s really a different user). The error details would help.

You can test this by hand in Windows File Explorer running as user YYY, trying to access user XXX’s files.

The --snapshot-policy=required will get access to locked files but won’t get past Windows access control. For that, the usual solution would be to run Duplicati as SYSTEM which is default for its Windows service, however I’m assuming YYY is not SYSTEM (otherwise why obscure?). It’s probably at least Administrator group, otherwise --snapshot-policy=required would have an error, but might not really be running elevated. Some people dislike running as SYSTEM because it’s too powerful. I don’t know if you have such reasons.

If you don’t in fact want the SharePoint files, you could exclude them. I’m not sure what the goal is besides NOT having 170 warnings. I’m also not sure if there are also warnings beyond what SharePoint files made.

dejhost · February 14, 2019, 9:31pm

Did this happen?

Of course - I’m not making things up. But since this error appeared only once, I suspect the reason might have something to do with the logfile not being in write-mode. Maybe the logfile was still open in a texteditor, or the texteditor didn’t close correctly…
However, I can access the logfile manually now. Going back to the timestamp, I see most errors relate to Sharepoint, Dropbox and googledrive:

2019-01-30 20:25:25 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\XXX\Dropbox (YYY).dropbox.cache\qebcobkqbgpbzfynfuwbof\placeholder.ai
System.IO.IOException: (1920) Das System kann auf die Datei nicht zugreifen: [\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy40\Users\XXX\Dropbox (YYY).dropbox.cache\qebcobkqbgpbzfynfuwbof\placeholder.ai]
bei Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
bei Alphaleonis.Win32.Filesystem.File.CreateFileCore(KernelTransaction transaction, String path, ExtendedFileAttributes attributes, FileSecurity fileSecurity, FileMode fileMode, FileSystemRights fileSystemRights, FileShare fileShare, Boolean checkPath, PathFormat pathFormat)
bei Alphaleonis.Win32.Filesystem.File.OpenCore(KernelTransaction transaction, String path, FileMode mode, FileSystemRights rights, FileShare share, ExtendedFileAttributes attributes, Nullable`1 bufferSize, FileSecurity security, PathFormat pathFormat)
bei Duplicati.Library.Snapshots.SystemIOWindows.FileOpenRead(String path)
bei Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.<>c__DisplayClass1_0.<b__0>d.MoveNext()

and

2019-01-30 20:25:17 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\YYY\Documents\XXX\Sharepoint\SharePoint\Admin - Documents
System.IO.IOException: (19) Das Medium ist schreibgeschützt: [\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy40\Users\YYY\Documents\XXX\Sharepoint\SharePoint\Admin - Documents*]
bei Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
bei Alphaleonis.Win32.Filesystem.FindFileSystemEntryInfo.FindFirstFile(String pathLp, WIN32_FIND_DATA& win32FindData)
bei Alphaleonis.Win32.Filesystem.FindFileSystemEntryInfo.d__31.MoveNext() bei System.Linq.Buffer1…ctor(IEnumerable1 source) bei System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
bei Duplicati.Library.Snapshots.WindowsSnapshot.ListFiles(String localFolderPath)
bei Duplicati.Library.Utility.Utility.d__23.MoveNext()

I’m happy to exclude the Sharepoint from backup, but my attempts failed for some reason earlier. In the configuration of duplicati => Source Data => Filters => Exclude Folders.
I excluded the corresponding folder just now once again, and it works: I receive “only” 105 warnings. Most (possibly all?) of them relate to googledrive and dropbox. Both should get backed up. Find below the error related to googledrive (the dropbox error is identical to the one from a few weeks ago):

2019-02-14 22:06:34 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\Users\XXX\Google Drive
System.UnauthorizedAccessException: (5) Zugriff verweigert: [\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy14\Users\XXX\Google Drive*]
bei Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
bei Alphaleonis.Win32.Filesystem.FindFileSystemEntryInfo.FindFirstFile(String pathLp, WIN32_FIND_DATA& win32FindData)
bei Alphaleonis.Win32.Filesystem.FindFileSystemEntryInfo.d__31.MoveNext() bei System.Linq.Buffer1…ctor(IEnumerable1 source) bei System.Linq.Enumerable.ToArray[TSource](IEnumerable1 source)
bei Duplicati.Library.Snapshots.WindowsSnapshot.ListFiles(String localFolderPath)
bei Duplicati.Library.Utility.Utility.d__23.MoveNext()

The --snapshot-policy=required will get access to locked files but won’t get past Windows access control. For that, the usual solution would be to run Duplicati as SYSTEM which is default for its Windows service, however I’m assuming YYY is not SYSTEM (otherwise why obscure?). It’s probably at least Administrator group, otherwise --snapshot-policy=required would have an error, but might not really be running elevated. Some people dislike running as SYSTEM because it’s too powerful. I don’t know if you have such reasons.

I’d be happy to run duplicati elevated. This is what the taskmanager shows:

Not sure what this means for my case.

If you don’t in fact want the SharePoint files, you could exclude them. I’m not sure what the goal is besides NOT having 170 warnings. I’m also not sure if there are also warnings beyond what SharePoint files made

Good point! As you mentioned yourself: sometimes errors/warnings are like symptoms of a real problem. 170 Warnings make it quite difficult to track (real) problems.

Here is my conclusion so far: Duplicati has problems accessing the files which are mirrored in the cloud. Snapshotpolicy=required does not do the trick, although duplicati is running as a SYSTEM. I think this is odd, since all files of googledrive, dropbox and sharepoint are problematic - although they are not accessed/synced, while duplicati is running its backup.

ts678 · February 15, 2019, 10:28pm

I’ve been looking into this awhile, and there’s more looking needed. I hope you can research and test.

is interesting for two reasons. One is that it’s in the cache, which (like most caches) is for temporary storage of data not meant for direct access which is automatically maintained and frequently cleared.

How to clear the cache folder

If that’s the end of it, then perhaps backing up cache isn’t necessary, however I suspect the sampling simply didn’t show the same problem in the main Dropbox folder. Problem’s clue is the placeholder.

I assume you have Dropbox Professional or Business with Smart Sync. If not, ignore the next portions.

Smart Sync
Smart Sync and system extensions
Going deeper with Project Infinite

I suspect it’s OK with a mirror, but not OK with cloud software that plays fancy tricks such as not doing actual mirroring (as older sync software likes to do), but on-demand on-line action (as newer ways do). One gets a mix of files that are on the computer, or not on the computer. Not on computer = no backup.

Carbon Copy Cloner 5 Release Notes

Carbon Copy Cloner 5.1.6

October 12, 2018

Improved the handling of Microsoft OneDrive, Google Drive File Stream and Dropbox placeholder files. Please note that if you’re using any of these services, files that are marked as “online only” cannot be backed up. Learn more here.

Limitations of online-only placeholder files

When you choose to have these services store your files only online, do so with the understanding that it’s not possible to maintain a local backup of those files.

I’m not sure I agree, but it does lead to potential trouble, such as downloading everything there to back up.

Prevent unwanted OneDrive file downloads

Cloud drive integration (files on-demand) took a survey of possible treatments. It’s not clear which is best.

Does this seem to fit so far? I haven’t seen any backup that deals with it well, which possibly could be due to an insufficient API for the backup software to ask for the file temporarily (then put things back as before) or perhaps because it’s hard to find a sensible way to do it at the client when all the files are in the cloud…

If OK so far, the next question is whether your Google Drive app is Drive File Stream, or Backup and Sync. Drive File Stream has the same issue. If Backup and Sync, right click Google Drive to make sure Security grants access by the SYSTEM user that you found in Task Manager. If not, access will need to be granted. When I first set up Google Drive backup, I had to do this. The new Backup and Sync seems better though.

dejhost · February 16, 2019, 11:14pm

Hello ts678 - thank you once again for all the time and thoughts you put into this!

I cleared the cache of dropbox (both, the private and the business version), and initiated a backup. 5 Warnings remain (so 100 warnings less than prior this change). In a next step, I excluded the dropbox cache folders from backup - this should do the trick in the long run.

The 5 remaining warnings all relate to the googledrive.
The tool in question is called “Backup and Sync from google”. In its preferences, “Sync my drive to this computer” is activated. So the files really are on my local harddrive. I can access them while beeing offline.
Right-click the googledrive-folder => Security => Add SYSTEM as user and grant “read” access.

At this point, I initiated another duplicati backup. And voila: I can now see duplicati transfering the files of the googledrive. Since I have no changes on my googledrive since the backup before, this filetransfer shows that the access was an issue.

Since googledrive hosts few gigabytes of data, I won’t be staying up until backup is finished. But I expect, that no more warnings will pop up.

Keep you posted!

dejhost · February 17, 2019, 3:15pm

Ok, we’re getting there:
2 Warnings remain. They refer to another user-account on the computer:

2019-02-17 15:36:41 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\XXX\AppData\Local\Microsoft\WindowsApps\MicrosoftEdge.exe
System.IO.IOException: (1920) Das System kann auf die Datei nicht zugreifen: [\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy31\Users\XXX\AppData\Local\Microsoft\WindowsApps\MicrosoftEdge.exe]
bei Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath)
bei Alphaleonis.Win32.Filesystem.File.CreateFileCore(KernelTransaction transaction, String path, ExtendedFileAttributes attributes, FileSecurity fileSecurity, FileMode fileMode, FileSystemRights fileSystemRights, FileShare fileShare, Boolean checkPath, PathFormat pathFormat)
bei Alphaleonis.Win32.Filesystem.File.OpenCore(KernelTransaction transaction, String path, FileMode mode, FileSystemRights rights, FileShare share, ExtendedFileAttributes attributes, Nullable1 bufferSize, FileSecurity security, PathFormat pathFormat) bei Duplicati.Library.Snapshots.SystemIOWindows.FileOpenRead(String path) bei Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.<>c__DisplayClass1_0.<<Run>b__0>d.MoveNext() 2019-02-17 15:36:41 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\XXX\AppData\Local\Microsoft\WindowsApps\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe System.IO.IOException: (1920) Das System kann auf die Datei nicht zugreifen: [\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy31\Users\XXX\AppData\Local\Microsoft\WindowsApps\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\MicrosoftEdge.exe] bei Alphaleonis.Win32.NativeError.ThrowException(UInt32 errorCode, String readPath, String writePath) bei Alphaleonis.Win32.Filesystem.File.CreateFileCore(KernelTransaction transaction, String path, ExtendedFileAttributes attributes, FileSecurity fileSecurity, FileMode fileMode, FileSystemRights fileSystemRights, FileShare fileShare, Boolean checkPath, PathFormat pathFormat) bei Alphaleonis.Win32.Filesystem.File.OpenCore(KernelTransaction transaction, String path, FileMode mode, FileSystemRights rights, FileShare share, ExtendedFileAttributes attributes, Nullable1 bufferSize, FileSecurity security, PathFormat pathFormat)
bei Duplicati.Library.Snapshots.SystemIOWindows.FileOpenRead(String path)
bei Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.<>c__DisplayClass1_0.<b__0>d.MoveNext()

The user SYSTEM has appropriate rights… however, I can very well live with these warnings. These files shouldn’t have been backed up anyway, so I will exclude them in the future. So for my part, all my questions are answered, and all problems solved.

I will put some notes into my wiki: Next time I install duplicati, I will have it run as SYSTEM right from the start. In case of errors/warnings, I will doublecheck that SYSTEM has the necessary rights.

Thank you for guiding me through this - errors and warnings during backups tend to make me nervous

ts678 · February 17, 2019, 9:25pm

I’m glad you got it working satisfactorily. Depending on how much more you want to work on improving anything, I’d like to at least try to clarify the situation, and maybe get deeper technically. Starting easier:

Although I hate to suggest breaking what’s been fixed, it would be nice to know if versions without Smart Sync could hit this, for example if cache can break even if Smart Sync isn’t available to the final user file.

OneDrive Files On-Demand had an earlier case of this issue, and Duplicati did some work like Fix 2829: Reparse points aren’t always symlinks #3014 to fix issue With Windows Fall Creators Update OneDriver Folder cannot be backupped any longer #2829. There were also some forum reports about this, I believe.

I almost never use OneDrive, so am not quite sure where things stand now in terms of what’s backed up. Regardless, it might be possible to get Dropbox to reach parity with whatever it is that OneDrive’s gotten. This wouldn’t be anything I would develop, but I could probably help with your research. Or we could wait. Possibly things will stay quiet on the Dropbox front for awhile. There’s ample other work to do meanwhile.

NTFS reparse point is the kind of thing this will get into. See earlier Dropbox technical article links as well. Fsutil reparsepoint is one tool that can query paths to see if some high-tech remote capability might exist. –exclude-files-attributes can exclude reparse points, but I’d worry that it might exclude desirable varieties.

That’s pretty deep, so no problems if you don’t want to dig in. To simply close this for now, I’d appreciate a note on what the SharePoint issue turned out to be (if you know), and perhaps a bit more OneDrive detail.

Thanks!

dejhost · February 21, 2019, 10:21am

Here is again the error message about OneDrive:

2019-01-25 11:50:09 +01 - [Warning-Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.FileEntry-PathProcessingFailed]: Failed to process path: C:\Users\XXX\OneDrive - YYY.849C9593-D756-4E56-8D6E-42412F2A707B
System.IO.IOException: The process cannot access the file ‘C:\Users\XXX\OneDrive - YYY.849C9593-D756-4E56-8D6E-42412F2A707B’ because it is being used by another process.
at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
at System.IO.FileStream…ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Duplicati.Library.Snapshots.SystemIOWindows.FileOpenRead(String path)
at Duplicati.Library.Main.Operation.Backup.FileBlockProcessor.<>c__DisplayClass1_0.<b__0>d.MoveNext()

I’m running the client “Microsoft OneDrive” version 19.002.0107.0008, which is set to mirror various folders. I am not sure how sharepoint is working, nor how it is set up, but I believe that sharepoint is using the onedrive-client for synchronization of files and folders.

I simply got rid of my problem by excluding

C:\Users\XXX\OneDrive - {company name]}
from the backup list. We have stopped using onedrive actively in our company anyway, and the files have been unchanged for many months. So the OneDrive cloud just serves as a form of archive. I am using the opensource solution syncthing today, and have not had any conflicts with duplicati.

So what you ask me to do is to install Smart Sync, and see if it creates problems?

ts678 · February 21, 2019, 12:39pm

Unclear. Not sure where we’ve been. Maybe an install, maybe settings, maybe theory was wrong. Theory:

Yet the next portions seemed to help. Was Smart Sync ever there? It could have been, given your level of Dropbox, however at installation time it looks like you would be led through installation of a minifilter driver which processes access to your file-which-is-maybe-not-local. You apparently need the desktop app too.

These integrations are core to current and future Dropbox client services. There is no way to disable or opt-out of using the system extension or minifilter driver on the Dropbox desktop client.

How to change desktop app preferences has Smart Sync. Smart Sync for team admins also weighs in…

Do you think you had Smart Sync before, and turned it off? Or were all the fixes done by doing excludes?

Looking at my reply here to your post on how you cleared the problem, the private and business versions proceeded in parallel, first with cache clears, then with excludes. This leaves the question of which of the versions was having trouble initially (maybe you recall from looking at the file lists). If personal had trouble then that may mean Dropbox is using streaming technology in an internal way such as cache, and if so it might have more widespread impact, but could also be examined more easily, e.g. in free Dropbox Basic.

Basically I’m just looking for any additional clues you have (or can readily get) by doing nothing more than backtracking a bit through your parallel fixes. You also have vastly more Dropbox experience. I have none. Simply wishing to characterize the problem a bit better before we leave, in hope it will help someone else.

dejhost · February 21, 2019, 3:15pm

I have installed dropbox v. 67.4.83, connecting to some sort of professional account. I do not use “Smart Sync”, and it has never been installed/activated. However, “Dropbox system extension” was turned on all the time. While writing this post, I turn it off.

Correct: After emptying the local dropbox-cache, 100 warnings disappeared. Only once this was clear, I excluded the cache-files from the backup. I assume the cache would fill up again after some time, and the errors would reappear…

Yes, I have also a private dropbox account, but when I logged into the business/professional account some months ago, the private account logged out. So I access my private account only through the webbrowser. Currently, the private account is not synchronized with my Pc.

Don’t hesitate to ask further questions in order to get the answers you’re looking for.