Possible Bug - Duplicati stops discovering files when a file is in use

My title says possible bug because it may just be I don’t understand what I’m doing wrong.

This is the version I’m running.: Duplicati - 2.0.4.5_beta_2018-11-28

This is what i’m trying to achieve, nightly backups of the my entire M drive with differentials going forward. This is what’s happening, I make my backup and choose my M drive, what happens is when duplicati is running it will discover everything up until it runs into an issue with error a file locked or file open. Then what happens is it stops discovering all my data on my M Drive.

Last successful backup:
Yesterday at 10:22 PM (took 00:53:08)
Run now
Next scheduled run:
Today at 1:00 PM
Source:
30.12 GB
Backup:
19.60 GB / 1 Version

The only additional option I have is to ignore system files.

As you can see only 19.60 out of 30.12 has been backed up. When I try run now duplicati doesn’t pick up the remaining 19.60. Any help would be greatly appreciated.

MainOperation: Test

Verifications: [

Key: duplicati-20190428T012941Z.dlist.zip.aes

Value: [],

Key: duplicati-i3b642c6160fa499a962d8d020c6a7c80.dindex.zip.aes

Value: [],

Key: duplicati-bda37469e69f748619bfc09edba86558b.dblock.zip.aes

Value: []

]

ParsedResult: Success

Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)

EndTime: 4/28/2019 11:17:05 AM (1556464625)

BeginTime: 4/28/2019 11:16:52 AM (1556464612)

Duration: 00:00:13.1794836

Messages: [

2019-04-28 11:16:52 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started,

2019-04-28 11:16:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),

2019-04-28 11:16:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (803 bytes),

2019-04-28 11:16:58 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20190428T012941Z.dlist.zip.aes (8.32 MB),

2019-04-28 11:16:58 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20190428T012941Z.dlist.zip.aes (8.32 MB),

]

Warnings:

Errors:

BackendStatistics:

RemoteCalls: 4

BytesUploaded: 0

BytesDownloaded: 61231959

FilesUploaded: 0

FilesDownloaded: 3

FilesDeleted: 0

FoldersCreated: 0

RetryAttempts: 0

UnknownFileSize: 0

UnknownFileCount: 0

KnownFileCount: 803

KnownFileSize: 21043221847

LastBackupDate: 4/27/2019 9:29:41 PM (1556414981)

BackupListCount: 1

TotalQuotaSpace: 0

FreeQuotaSpace: 0

AssignedQuotaSpace: -1

ReportedQuotaError: False

ReportedQuotaWarning: False

ParsedResult: Success

Version: 2.0.4.5 (2.0.4.5_beta_2018-11-28)

Messages: [

    2019-04-28 11:16:52 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Test has started,

    2019-04-28 11:16:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  (),

    2019-04-28 11:16:57 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (803 bytes),

    2019-04-28 11:16:58 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20190428T012941Z.dlist.zip.aes (8.32 MB),

    2019-04-28 11:16:58 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20190428T012941Z.dlist.zip.aes (8.32 MB),

]

Warnings: []

Errors: []

Hello @phyregold and welcome to the forum!

These numbers are not comparable, so inequality is normal. It doesn’t prove a problem (or lack of one).

The backup number is the size of the remote backup, and can be less than the source file sizes due to compression, or greater than the source file sizes due to multiple versions being saved onto the remote. Duplicati builds the next backup on the current one, storing only the changes, however these do build up and can eventually lead to a backup with many versions and much space. Retention rules can limit that.

You show BackendStatistics from your backup job log. At the top of the log, do values for ExaminedFiles and SizeOfExaminedFiles seem right, for example compared to what File Explorer or similar are saying?

I guess it’s right. I’ll try a complete restore for confidence sake.

Did the restore come out about right? If you REALLY want to see if it’s right, you can use something like WinMerge right after backup and a restore, to see if the restore picked up all the files that it should have. Default restore will optimize speed on obtaining the file data from local file blocks if it can, but I think that doesn’t matter here because you’re wondering if the file is backed up at all, without a worry on contents. For very best simulation of a disaster recovery, you can try a direct restore to a different Windows PC…

Duplicati skipping lots of files/folders reminded me of this one, but that’s on individual files gone missing.