Conflict when changed to smart retention and can't revert either!

Jay_T · January 28, 2019, 4:33am

I originally created 2 regular backup schedules, both using retention policy of ‘keep xx number of backups’. After a recent update (2.0.2 or something?) I edited both to the ‘Smart backup retention’ option, but this caused an error:
" Failed: Setting multiple retention options (–keep-versions, --retention-policy) is not permitted

Details: Duplicati.Library.Interface.UserInformationException: Setting multiple retention options (–keep-versions, --retention-policy) is not permitted

at Duplicati.Library.Main.Controller.ValidateOptions()

at Duplicati.Library.Main.Controller.SetupCommonOptions(ISetCommonOptions result, String& paths, IFilter& filter)

at Duplicati.Library.Main.Controller.RunAction[T](T result, String& paths, IFilter& filter, Action`1 method) "
(quoting whole email report)
So i thought I would just change it back, but it doesn’t like that either, same error. Now it is running through most of the files and jamming up with 51 or 52 files to go.
Tried changing to ‘keep 1 copy’ and that sometimes works , tried ‘compacting’, tried repairing and recreating database. What else do I try???
I’m using B2 cloud storage. Should I empty the bucket and start from scratch? I’ve tried adding a new bucket and directing the backup to go in there instead, which by my thinking should make it a fresh start but that didn’t work either. Tried upgrading from beta to 2.0.4.12_canary.
Would value anyone’s help!
Screenshot of the current clogged process if it helps:

Jay_T · January 28, 2019, 7:02pm

UPDATE: The backups are completing successfully - when I give them time! The small one used to take about 1 minute and the large one about 13 minutes.
I would like to know why the small 100MB backup is now taking 3 hours though??

ts678 · January 28, 2019, 10:10pm

You can watch the action (or lack of action) at About → Show log → Live. Using Information level is a nice start. If you want Profiling, it might be better to set that –log-file-log-level and accumulate an actual –log-file.

Even looking at the UI may help, e.g. see the bottom of your original screen shot. Where’s it at for 3 hours? Running the backup using its Commandline gives a more linear text view. You can set a –console-log-level.

JonMikelV · January 28, 2019, 11:41pm

Hi @Jay_T, welcome to the forum!

Most likely the longer duration is temporary and related to compacting / cleanup up the “out of retention rules” files.

Check the job log for the long running job and see if there’s a “compacting” type section.

If it’s still running long, try disabled auto-compacting to test if that is the source of the slowdown.

Jay_T · January 29, 2019, 12:21am

Thanks @ts678 . OK, used the live log for this backup, started again 10 mins ago right on schedule:

I don’t understand everything I see here; is there enough info for you to interpret? Note that “\SERVER\Data (E)” is also a mapped network drive “S:” - I trust that is not an issue?

ts678 · January 29, 2019, 12:43am

Reading in reverse order, the temporary file deletion and comment about synthetic filelist indicate that the previous backup was interrupted. The fact that it mentions a dindex file rather than expected dlist may be caused by a suspected bug. The change journal comment makes me wonder if --usn-policy option is set even though it only works on local NTFS drives (not mapped network drives). You could try removing that.

The big question is unresolved. If it’s just sitting there showing no more, maybe raise log level to Profiling?

SMB sometimes does weird things, especially if Duplicati is a Windows service (which is not the default). Maybe I should have asked for Verbose so you could at least see one line per file as it scans source area, however I’m not sure if it would have announced that 1409 file count if it hadn’t actually looked over files…

Jay_T · January 29, 2019, 12:56am

Hello @JonMikelV,
Good suggestions; but it does show there is only 3 (new) versions there now, as expected. The compacting only took 0.04 seconds according to the log:

> {
>   "DeletedFiles": 1,
>   "DeletedFolders": 0,
>   "ModifiedFiles": 1338,
>   "ExaminedFiles": 1357,
>   "OpenedFiles": 1341,
>   "AddedFiles": 3,
>   "SizeOfModifiedFiles": 1789952,
>   "SizeOfAddedFiles": 64204,
>   "SizeOfExaminedFiles": 174120874,
>   "SizeOfOpenedFiles": 155075782,
>   "NotProcessedFiles": 0,
>   "AddedFolders": 0,
>   "TooLargeFiles": 0,
>   "FilesWithError": 0,
>   "ModifiedFolders": 0,
>   "ModifiedSymlinks": 0,
>   "AddedSymlinks": 0,
>   "DeletedSymlinks": 0,
>   "PartialBackup": false,
>   "Dryrun": false,
>   "MainOperation": "Backup",
>   "CompactResults": {
>     "DeletedFileCount": 0,
>     "DownloadedFileCount": 0,
>     "UploadedFileCount": 0,
>     "DeletedFileSize": 0,
>     "DownloadedFileSize": 0,
>     "UploadedFileSize": 0,
>     "Dryrun": false,
>     "MainOperation": "Compact",
>     "ParsedResult": "Success",
>     "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>     "EndTime": "2019-01-28T23:06:42.280491Z",
>     "BeginTime": "2019-01-28T23:06:42.2391247Z",
>     "Duration": "00:00:00.0413663",
>     "MessagesActualLength": 0,
>     "WarningsActualLength": 0,
>     "ErrorsActualLength": 0,
>     "Messages": null,
>     "Warnings": null,
>     "Errors": null,
>     "BackendStatistics": {
>       "RemoteCalls": 11,
>       "BytesUploaded": 285559,
>       "BytesDownloaded": 52794366,
>       "FilesUploaded": 3,
>       "FilesDownloaded": 6,
>       "FilesDeleted": 0,
>       "FoldersCreated": 0,
>       "RetryAttempts": 0,
>       "UnknownFileSize": 0,
>       "UnknownFileCount": 0,
>       "KnownFileCount": 11,
>       "KnownFileSize": 103023967,
>       "LastBackupDate": "2019-01-29T09:00:00+13:00",
>       "BackupListCount": 3,
>       "TotalQuotaSpace": 0,
>       "FreeQuotaSpace": 0,
>       "AssignedQuotaSpace": -1,
>       "ReportedQuotaError": false,
>       "ReportedQuotaWarning": false,
>       "MainOperation": "Backup",
>       "ParsedResult": "Success",
>       "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>       "EndTime": "0001-01-01T00:00:00",
>       "BeginTime": "2019-01-28T20:00:00.0089544Z",
>       "Duration": "00:00:00",
>       "MessagesActualLength": 0,
>       "WarningsActualLength": 0,
>       "ErrorsActualLength": 0,
>       "Messages": null,
>       "Warnings": null,
>       "Errors": null
>     }
>   },
>   "DeleteResults": {
>     "DeletedSetsActualLength": 0,
>     "DeletedSets": [],
>     "Dryrun": false,
>     "MainOperation": "Delete",
>     "CompactResults": {
>       "DeletedFileCount": 0,
>       "DownloadedFileCount": 0,
>       "UploadedFileCount": 0,
>       "DeletedFileSize": 0,
>       "DownloadedFileSize": 0,
>       "UploadedFileSize": 0,
>       "Dryrun": false,
>       "MainOperation": "Compact",
>       "ParsedResult": "Success",
>       "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>       "EndTime": "2019-01-28T23:06:42.280491Z",
>       "BeginTime": "2019-01-28T23:06:42.2391247Z",
>       "Duration": "00:00:00.0413663",
>       "MessagesActualLength": 0,
>       "WarningsActualLength": 0,
>       "ErrorsActualLength": 0,
>       "Messages": null,
>       "Warnings": null,
>       "Errors": null,
>       "BackendStatistics": {
>         "RemoteCalls": 11,
>         "BytesUploaded": 285559,
>         "BytesDownloaded": 52794366,
>         "FilesUploaded": 3,
>         "FilesDownloaded": 6,
>         "FilesDeleted": 0,
>         "FoldersCreated": 0,
>         "RetryAttempts": 0,
>         "UnknownFileSize": 0,
>         "UnknownFileCount": 0,
>         "KnownFileCount": 11,
>         "KnownFileSize": 103023967,
>         "LastBackupDate": "2019-01-29T09:00:00+13:00",
>         "BackupListCount": 3,
>         "TotalQuotaSpace": 0,
>         "FreeQuotaSpace": 0,
>         "AssignedQuotaSpace": -1,
>         "ReportedQuotaError": false,
>         "ReportedQuotaWarning": false,
>         "MainOperation": "Backup",
>         "ParsedResult": "Success",
>         "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>         "EndTime": "0001-01-01T00:00:00",
>         "BeginTime": "2019-01-28T20:00:00.0089544Z",
>         "Duration": "00:00:00",
>         "MessagesActualLength": 0,
>         "WarningsActualLength": 0,
>         "ErrorsActualLength": 0,
>         "Messages": null,
>         "Warnings": null,
>         "Errors": null
>       }
>     },
>     "ParsedResult": "Success",
>     "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>     "EndTime": "2019-01-28T23:06:42.280491Z",
>     "BeginTime": "2019-01-28T23:06:41.4807346Z",
>     "Duration": "00:00:00.7997564",
>     "MessagesActualLength": 0,
>     "WarningsActualLength": 0,
>     "ErrorsActualLength": 0,
>     "Messages": null,
>     "Warnings": null,
>     "Errors": null,
>     "BackendStatistics": {
>       "RemoteCalls": 11,
>       "BytesUploaded": 285559,
>       "BytesDownloaded": 52794366,
>       "FilesUploaded": 3,
>       "FilesDownloaded": 6,
>       "FilesDeleted": 0,
>       "FoldersCreated": 0,
>       "RetryAttempts": 0,
>       "UnknownFileSize": 0,
>       "UnknownFileCount": 0,
>       "KnownFileCount": 11,
>       "KnownFileSize": 103023967,
>       "LastBackupDate": "2019-01-29T09:00:00+13:00",
>       "BackupListCount": 3,
>       "TotalQuotaSpace": 0,
>       "FreeQuotaSpace": 0,
>       "AssignedQuotaSpace": -1,
>       "ReportedQuotaError": false,
>       "ReportedQuotaWarning": false,
>       "MainOperation": "Backup",
>       "ParsedResult": "Success",
>       "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>       "EndTime": "0001-01-01T00:00:00",
>       "BeginTime": "2019-01-28T20:00:00.0089544Z",
>       "Duration": "00:00:00",
>       "MessagesActualLength": 0,
>       "WarningsActualLength": 0,
>       "ErrorsActualLength": 0,
>       "Messages": null,
>       "Warnings": null,
>       "Errors": null
>     }
>   },
>   "RepairResults": null,
>   "TestResults": {
>     "MainOperation": "Test",
>     "VerificationsActualLength": 6,
>     "Verifications": [
>       {
>         "Key": "duplicati-20190128T200000Z.dlist.zip.aes",
>         "Value": []
>       },
>       {
>         "Key": "duplicati-20190128T041527Z.dlist.zip.aes",
>         "Value": []
>       },
>       {
>         "Key": "duplicati-i721bbd9876154fecaf6434533758bcee.dindex.zip.aes",
>         "Value": []
>       },
>       {
>         "Key": "duplicati-if8429427de1e489d88927fa58ff2d84a.dindex.zip.aes",
>         "Value": []
>       },
>       {
>         "Key": "duplicati-b5bee105ac131482bac836cd54a4dbc5d.dblock.zip.aes",
>         "Value": []
>       },
>       {
>         "Key": "duplicati-b9522954b15c74218bb6c899315c3b53e.dblock.zip.aes",
>         "Value": []
>       }
>     ],
>     "ParsedResult": "Success",
>     "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>     "EndTime": "2019-01-28T23:07:31.5647835Z",
>     "BeginTime": "2019-01-28T23:06:45.7216892Z",
>     "Duration": "00:00:45.8430943",
>     "MessagesActualLength": 0,
>     "WarningsActualLength": 0,
>     "ErrorsActualLength": 0,
>     "Messages": null,
>     "Warnings": null,
>     "Errors": null,
>     "BackendStatistics": {
>       "RemoteCalls": 11,
>       "BytesUploaded": 285559,
>       "BytesDownloaded": 52794366,
>       "FilesUploaded": 3,
>       "FilesDownloaded": 6,
>       "FilesDeleted": 0,
>       "FoldersCreated": 0,
>       "RetryAttempts": 0,
>       "UnknownFileSize": 0,
>       "UnknownFileCount": 0,
>       "KnownFileCount": 11,
>       "KnownFileSize": 103023967,
>       "LastBackupDate": "2019-01-29T09:00:00+13:00",
>       "BackupListCount": 3,
>       "TotalQuotaSpace": 0,
>       "FreeQuotaSpace": 0,
>       "AssignedQuotaSpace": -1,
>       "ReportedQuotaError": false,
>       "ReportedQuotaWarning": false,
>       "MainOperation": "Backup",
>       "ParsedResult": "Success",
>       "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>       "EndTime": "0001-01-01T00:00:00",
>       "BeginTime": "2019-01-28T20:00:00.0089544Z",
>       "Duration": "00:00:00",
>       "MessagesActualLength": 0,
>       "WarningsActualLength": 0,
>       "ErrorsActualLength": 0,
>       "Messages": null,
>       "Warnings": null,
>       "Errors": null
>     }
>   },
>   "ParsedResult": "Success",
>   "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>   "EndTime": "2019-01-28T23:07:32.492513Z",
>   "BeginTime": "2019-01-28T20:00:00.0089544Z",
>   "Duration": "03:07:32.4835586",
>   "MessagesActualLength": 26,
>   "WarningsActualLength": 0,
>   "ErrorsActualLength": 0,
>   "Messages": [
>     "2019-01-29 09:00:00 +13 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
>     "2019-01-29 09:00:00 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
>     "2019-01-29 09:00:04 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (8 bytes)",
>     "2019-01-29 09:00:04 +13 - [Information-Duplicati.Library.Main.Operation.BackupHandler-FailedToUseChangeJournal]: Failed to use change journal for volume \"\\\\SERVER\\Data (E)\": The system cannot find the path specified",
>     "2019-01-29 12:06:27 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b5bee105ac131482bac836cd54a4dbc5d.dblock.zip.aes (154.58 KB)",
>     "2019-01-29 12:06:37 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b5bee105ac131482bac836cd54a4dbc5d.dblock.zip.aes (154.58 KB)",
>     "2019-01-29 12:06:38 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i721bbd9876154fecaf6434533758bcee.dindex.zip.aes (2.42 KB)",
>     "2019-01-29 12:06:39 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i721bbd9876154fecaf6434533758bcee.dindex.zip.aes (2.42 KB)",
>     "2019-01-29 12:06:40 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)",
>     "2019-01-29 12:06:41 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)",
>     "2019-01-29 12:06:42 +13 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted",
>     "2019-01-29 12:06:42 +13 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required",
>     "2019-01-29 12:06:42 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
>     "2019-01-29 12:06:45 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (11 bytes)",
>     "2019-01-29 12:06:45 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)",
>     "2019-01-29 12:06:49 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)",
>     "2019-01-29 12:06:49 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20190128T041527Z.dlist.zip.aes (121.72 KB)",
>     "2019-01-29 12:06:50 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20190128T041527Z.dlist.zip.aes (121.72 KB)",
>     "2019-01-29 12:06:50 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i721bbd9876154fecaf6434533758bcee.dindex.zip.aes (2.42 KB)",
>     "2019-01-29 12:06:53 +13 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i721bbd9876154fecaf6434533758bcee.dindex.zip.aes (2.42 KB)"
>   ],
>   "Warnings": [],
>   "Errors": [],
>   "BackendStatistics": {
>     "RemoteCalls": 11,
>     "BytesUploaded": 285559,
>     "BytesDownloaded": 52794366,
>     "FilesUploaded": 3,
>     "FilesDownloaded": 6,
>     "FilesDeleted": 0,
>     "FoldersCreated": 0,
>     "RetryAttempts": 0,
>     "UnknownFileSize": 0,
>     "UnknownFileCount": 0,
>     "KnownFileCount": 11,
>     "KnownFileSize": 103023967,
>     "LastBackupDate": "2019-01-29T09:00:00+13:00",
>     "BackupListCount": 3,
>     "TotalQuotaSpace": 0,
>     "FreeQuotaSpace": 0,
>     "AssignedQuotaSpace": -1,
>     "ReportedQuotaError": false,
>     "ReportedQuotaWarning": false,
>     "MainOperation": "Backup",
>     "ParsedResult": "Success",
>     "Version": "2.0.4.12 (2.0.4.12_canary_2019-01-16)",
>     "EndTime": "0001-01-01T00:00:00",
>     "BeginTime": "2019-01-28T20:00:00.0089544Z",
>     "Duration": "00:00:00",
>     "MessagesActualLength": 0,
>     "WarningsActualLength": 0,
>     "ErrorsActualLength": 0,
>     "Messages": null,
>     "Warnings": null,
>     "Errors": null
>   }
> }

Jay_T · January 29, 2019, 2:26am

Some of the previous backups were interrupted by me, but not the last one.
Have checked, --usn policy is not set.
See current log at Profiling level, 3 lines is sufficient but it continues like this ad infinitum:

Seems to have got quite stuck this time.

Not a Windows service, just the default installation. I think 1409 would be a fair number for the file count.

JonMikelV · January 29, 2019, 3:04am

Thanks for the log, it does indeed show me theory isn’t the right one.

BTW, I edited your post by adding ~~~ json before and ~~~ after the logs to make them easier to read.

Jay_T · January 29, 2019, 3:53am

@JonMikelV , thanks for the edit and the handy tip!
Noting above comment about previous backups interrupted, I have allowed the last 3 to run right through. The latest one ran right on the tail of the previous one which did throw the same synthetic filelist warning again. And what do you know, the last one only took 1m 39s and no warnings - so it may have fixed itself!
Posting the Information level log here that covers most of the action, in case it helps anyone. I’ll see how both backups go overnight :

* Jan 29, 2019 4:35 PM: The operation Backup has completed

* Jan 29, 2019 4:35 PM: Backend event: Get - Completed: duplicati-b9522954b15c74218bb6c899315c3b53e.dblock.zip.aes (49.91 MB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-b9522954b15c74218bb6c899315c3b53e.dblock.zip.aes (49.91 MB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Completed: duplicati-b708a3567db7a4ee99e5bc7d0f309031f.dblock.zip.aes (45.55 MB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-b708a3567db7a4ee99e5bc7d0f309031f.dblock.zip.aes (45.55 MB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Completed: duplicati-icb25c32c60cb48179e76b6e236564f89.dindex.zip.aes (110.04 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-icb25c32c60cb48179e76b6e236564f89.dindex.zip.aes (110.04 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Completed: duplicati-i85ba107742f34028b23ff39cea4ec407.dindex.zip.aes (84.14 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-i85ba107742f34028b23ff39cea4ec407.dindex.zip.aes (84.14 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Completed: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Completed: duplicati-20190128T041527Z.dlist.zip.aes (121.72 KB)

* Jan 29, 2019 4:34 PM: Backend event: Get - Started: duplicati-20190128T041527Z.dlist.zip.aes (121.72 KB)

* Jan 29, 2019 4:34 PM: Backend event: List - Completed: (14 bytes)

* Jan 29, 2019 4:34 PM: Backend event: List - Started: ()

* Jan 29, 2019 4:34 PM: No remote filesets were deleted

* Jan 29, 2019 4:33 PM: Backend event: List - Completed: (14 bytes)

* Jan 29, 2019 4:33 PM: Backend event: List - Started: ()

* Jan 29, 2019 4:33 PM: The operation Backup has started

* Jan 29, 2019 4:33 PM: The operation Backup has completed

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-b785782b5125f4f2783ab2f0e34dc53b1.dblock.zip.aes (2.04 MB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-b785782b5125f4f2783ab2f0e34dc53b1.dblock.zip.aes (2.04 MB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-bd8912e51d6604bbe8aba6e0d35053448.dblock.zip.aes (3.32 MB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-bd8912e51d6604bbe8aba6e0d35053448.dblock.zip.aes (3.32 MB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-if8429427de1e489d88927fa58ff2d84a.dindex.zip.aes (53.25 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-if8429427de1e489d88927fa58ff2d84a.dindex.zip.aes (53.25 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-ia90b569ef9f24d71928f4e473ae95199.dindex.zip.aes (12.37 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-ia90b569ef9f24d71928f4e473ae95199.dindex.zip.aes (12.37 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-20190128T200000Z.dlist.zip.aes (121.87 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Completed: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:33 PM: Backend event: Get - Started: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:33 PM: Backend event: List - Completed: (14 bytes)

* Jan 29, 2019 4:33 PM: Backend event: List - Started: ()

* Jan 29, 2019 4:33 PM: No remote filesets were deleted

* Jan 29, 2019 4:33 PM: Backend event: Put - Completed: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:33 PM: Backend event: Put - Started: duplicati-20190129T033234Z.dlist.zip.aes (122.03 KB)

* Jan 29, 2019 4:33 PM: Backend event: Put - Completed: duplicati-ia90b569ef9f24d71928f4e473ae95199.dindex.zip.aes (12.37 KB)

* Jan 29, 2019 4:33 PM: Backend event: Put - Started: duplicati-ia90b569ef9f24d71928f4e473ae95199.dindex.zip.aes (12.37 KB)

* Jan 29, 2019 4:33 PM: Backend event: Put - Completed: duplicati-bd8912e51d6604bbe8aba6e0d35053448.dblock.zip.aes (3.32 MB)

* Jan 29, 2019 4:33 PM: Backend event: Put - Started: duplicati-bd8912e51d6604bbe8aba6e0d35053448.dblock.zip.aes (3.32 MB)

Jay_T · January 30, 2019, 2:33am

Smaller backup seems to be running fine now. Larger one seems to be rather slow and having problems with it’s filters… may I ask: is this the correct syntax for filters using the web-GUI?
Because it seems to be spending a lot of time going through this very folder, which is meant to be excluded!

Tried a database repair and it seemed to think the database was all good already

ts678 · January 30, 2019, 11:22pm

I’m not a filter expert, but having tried to learn them lately I think the Filters article is aimed at command line and special cases, while the GUI’s Creating a new backup job is too vague on translations intended to help. For example, I think the intent there is you type in a folder, and the GUI adds * wildcards and a trailing slash. You can see what it made using the three-dot menu, and you might also notice some bugs where GUI may need to be told a few times (via dropdown) what you want it to do. Keep at it. You can look in Commandline to see what sort of --exclude the GUI built. I think yours looks right for that setting, but not for input to GUI…

Problems using test-filter gets in some of this, while also giving ways for you to test filters, without backup.

Jay_T · January 31, 2019, 3:38am

OK, thanks @ts678
I’ll have a few tries at this. I had the complete folder address (S:\Jaden\Des -Old…) but just to compound the problem, my Windows 10 seems to drop it’s connections to all it’s mapped network drives overnight occasionally and suddenly all shortcuts with “S:” won’t work. Rather annoying and requires a reboot to fix it whenever it happens.
Don’t know if perhaps Duplicati is smarter and knows it’s way around this and always uses full addresses or not??

ts678 · January 31, 2019, 3:50am

If you desire source UNC paths instead of drive letters, I think they work, but sometimes authentication has trouble, especially when Duplicati is a Windows service instead of running as a Tray Icon as a normal user.

I’m not sure what sort of smartness you hoped for. I think if you configure Duplicati using S: it expects S:…

Jay_T · February 3, 2019, 9:59pm

Yes, I wondered Duplicati if would use the corresponding UNC path if the Windows drive letter stopped working. Does seem a ridiculous question but we are dealing with Windows:roll_eyes:

Anyway, I have run this larger backup successfully, 4 times, last took only 25 minutes. So I thought I’ll try change it to ‘Smart backup retention’ again. Was not happy, it now throws the original error again. Studied the command-line UI options again and spotted that it includes --keep-versions AND --retention-policy as separate options, hadn’t realised that before. This is (hopefully) exactly the solution I’m looking for!
It will now remove it’s conflicting saved options data and will be able to function as intended.
Thanks for putting me on to using the commandline tool!

Jay_T · February 3, 2019, 10:56pm

No, it doesn’t quite just do that How can I make it FORGET about the previous retention policy and just use the new one?
Works in the commandline but not as a normal scheduled job in the GUI. Do I need to export and edit and import the job somehow?

ts678 · February 4, 2019, 3:39am

Probably need to find it and delete it. Assuming you wish to control it all with the dropdown, make sure that About → Settings and the job’s screen 5 Advanced options have no –retention-policy or –keep-versions or
–keep-time, and (in theory) the dropdown should not allow you to set more than one of its choices at once.

You should be able to see the merging of the global settings and job options using Export as Commandline or Commandline (or run the backup and see if it complains). I think Export to File shows only job’s options.

You could try removing every one of those three options you can see anywhere (remove dropdown option by choosing “Keep all backups” from the dropdown), then see if somehow there’s still one of the three left.
Once everything is gone, try the dropdown again. If need be, we can look in databases, but let’s try UI first.

Jay_T · February 6, 2019, 7:03pm

Thanks @ts678. Yes, that’s basically what I thought, but it didn’t seem to quite work. What HAS worked is exporting the backup job, deleting it and importing it again and changing it to Smart retention policy before running it. I did have to set an advanced option to make it use the same local database path, but other than that it worked fine, and now only takes 12 minutes!