Deleting files - excessive download

Support
1

Hi,

I have been using duplicati for close to a year now.
My storage is on B2 and currently I am consuming 1.51 TB.
My backup runs daily and for the last year I have a maximum of 4 GB download a month. And I never have to retrieve.
Since Jan 14 - I noticed duplicati exceeded the quota of 11 GB a day. Today I monitored it and I noticed that it is stuck on “Deleting unwanted files” and even 14 GB were not enough.

Can you please advice what can I do? B2 charges on download. The behavior is extremely odd to me.

It is worth it to mention that I don’t put that much data at once. Rarely it happens.

For example -

Jan 12
Source Files Examined 123310 (1.51 TB) Opened 16 (23.99 MB) Added 10 (3.51 MB) Modified 6 (20.48 MB) Deleted 1

Jan 13
Source Files Examined 123312 (1.51 TB) Opened 14 (22.87 MB) Added 8 (1.59 MB) Modified 6 (21.28 MB) Deleted 6

Then Jan 14 - it error because it exceed the quota

Jan 15 - first attempt (it did ask me to repair)

{
  "MainOperation": "Repair",
  "RecreateDatabaseResults": null,
  "ParsedResult": "Success",
  "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
  "EndTime": "2023-01-15T20:23:08.578005Z",
  "BeginTime": "2023-01-15T20:22:08.667371Z",
  "Duration": "00:00:59.9106340",
  "MessagesActualLength": 11,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2023-01-15 12:22:08 -08 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Repair has started",
    "2023-01-15 12:22:12 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2023-01-15 12:23:04 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (67.04 KB)",
    "2023-01-15 12:23:07 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ib7a46085ec8a4765a32c63371531da2a.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ib7a46085ec8a4765a32c63371531da2a.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i991c65d456da4fee990a6cdc4f1aa419.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i991c65d456da4fee990a6cdc4f1aa419.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i2cf41c18ed2c47edbf97485199346a7d.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i2cf41c18ed2c47edbf97485199346a7d.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-id1286e5e2e1446f69b4623bbeaea938c.dindex.zip.aes (541 bytes)",
    "2023-01-15 12:23:08 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-id1286e5e2e1446f69b4623bbeaea938c.dindex.zip.aes (541 bytes)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 5,
    "BytesUploaded": 2164,
    "BytesDownloaded": 0,
    "FilesUploaded": 4,
    "FilesDownloaded": 0,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 68652,
    "KnownFileSize": 1795695727036,
    "LastBackupDate": "2023-01-14T12:00:04-08:00",
    "BackupListCount": 14,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Repair",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-01-15T20:22:08.66739Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Jan 15 - second attempt (I did manually cancel this one when it approached 14 GB download - after completing the current file)

{
  "DeletedFiles": 57,
  "DeletedFolders": 0,
  "ModifiedFiles": 8,
  "ExaminedFiles": 123338,
  "OpenedFiles": 79,
  "AddedFiles": 71,
  "SizeOfModifiedFiles": 9942387,
  "SizeOfAddedFiles": 136432020,
  "SizeOfExaminedFiles": 1661731240758,
  "SizeOfOpenedFiles": 146374407,
  "NotProcessedFiles": 0,
  "AddedFolders": 0,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 0,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 0,
    "DownloadedFileCount": 0,
    "UploadedFileCount": 0,
    "DeletedFileSize": 0,
    "DownloadedFileSize": 0,
    "UploadedFileSize": 0,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-01-15T20:41:15.209857Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 993,
      "BytesUploaded": 4120731731,
      "BytesDownloaded": 14612072859,
      "FilesUploaded": 159,
      "FilesDownloaded": 279,
      "FilesDeleted": 553,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 68262,
      "KnownFileSize": 1785338349950,
      "LastBackupDate": "2023-01-15T12:23:50-08:00",
      "BackupListCount": 14,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-15T20:23:50.094811Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 1,
    "DeletedSets": [
      {
        "Item1": 7,
        "Item2": "2023-01-08T12:00:04-08:00"
      }
    ],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": {
      "DeletedFileCount": 0,
      "DownloadedFileCount": 0,
      "UploadedFileCount": 0,
      "DeletedFileSize": 0,
      "DownloadedFileSize": 0,
      "UploadedFileSize": 0,
      "Dryrun": false,
      "VacuumResults": null,
      "MainOperation": "Compact",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-15T20:41:15.209857Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null,
      "BackendStatistics": {
        "RemoteCalls": 993,
        "BytesUploaded": 4120731731,
        "BytesDownloaded": 14612072859,
        "FilesUploaded": 159,
        "FilesDownloaded": 279,
        "FilesDeleted": 553,
        "FoldersCreated": 0,
        "RetryAttempts": 0,
        "UnknownFileSize": 0,
        "UnknownFileCount": 0,
        "KnownFileCount": 68262,
        "KnownFileSize": 1785338349950,
        "LastBackupDate": "2023-01-15T12:23:50-08:00",
        "BackupListCount": 14,
        "TotalQuotaSpace": 0,
        "FreeQuotaSpace": 0,
        "AssignedQuotaSpace": -1,
        "ReportedQuotaError": false,
        "ReportedQuotaWarning": false,
        "MainOperation": "Backup",
        "ParsedResult": "Success",
        "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
        "EndTime": "0001-01-01T00:00:00",
        "BeginTime": "2023-01-15T20:23:50.094811Z",
        "Duration": "00:00:00",
        "MessagesActualLength": 0,
        "WarningsActualLength": 0,
        "ErrorsActualLength": 0,
        "Messages": null,
        "Warnings": null,
        "Errors": null
      }
    },
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-01-15T21:20:32.210895Z",
    "BeginTime": "2023-01-15T20:39:56.457404Z",
    "Duration": "00:40:35.7534910",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 993,
      "BytesUploaded": 4120731731,
      "BytesDownloaded": 14612072859,
      "FilesUploaded": 159,
      "FilesDownloaded": 279,
      "FilesDeleted": 553,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 68262,
      "KnownFileSize": 1785338349950,
      "LastBackupDate": "2023-01-15T12:23:50-08:00",
      "BackupListCount": 14,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-15T20:23:50.094811Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 0,
    "Verifications": [],
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-01-15T21:21:23.633617Z",
    "BeginTime": "2023-01-15T21:21:22.407419Z",
    "Duration": "00:00:01.2261980",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 993,
      "BytesUploaded": 4120731731,
      "BytesDownloaded": 14612072859,
      "FilesUploaded": 159,
      "FilesDownloaded": 279,
      "FilesDeleted": 553,
      "FoldersCreated": 0,
      "RetryAttempts": 0,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 68262,
      "KnownFileSize": 1785338349950,
      "LastBackupDate": "2023-01-15T12:23:50-08:00",
      "BackupListCount": 14,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-15T20:23:50.094811Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Success",
  "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
  "EndTime": "2023-01-15T21:21:23.634128Z",
  "BeginTime": "2023-01-15T20:23:50.094805Z",
  "Duration": "00:57:33.5393230",
  "MessagesActualLength": 1997,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2023-01-15 12:23:50 -08 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
    "2023-01-15 12:25:05 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2023-01-15 12:25:52 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (67.05 KB)",
    "2023-01-15 12:36:50 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b8ccdf5342e6b4796ad3eed433270ad5c.dblock.zip.aes (49.91 MB)",
    "2023-01-15 12:36:58 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-beb928e3c734f43e89e25f1ad4a6ee354.dblock.zip.aes (49.93 MB)",
    "2023-01-15 12:37:10 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bfd63a78f93e04350b78ab62bcd335305.dblock.zip.aes (11.03 MB)",
    "2023-01-15 12:37:37 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b8ccdf5342e6b4796ad3eed433270ad5c.dblock.zip.aes (49.91 MB)",
    "2023-01-15 12:37:37 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-iade99aff69914c308f74907ea5698625.dindex.zip.aes (52.31 KB)",
    "2023-01-15 12:37:37 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-iade99aff69914c308f74907ea5698625.dindex.zip.aes (52.31 KB)",
    "2023-01-15 12:37:51 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-beb928e3c734f43e89e25f1ad4a6ee354.dblock.zip.aes (49.93 MB)",
    "2023-01-15 12:37:51 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ie70c4272c2c9403d859b756f5d1de0ee.dindex.zip.aes (70.59 KB)",
    "2023-01-15 12:37:51 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie70c4272c2c9403d859b756f5d1de0ee.dindex.zip.aes (70.59 KB)",
    "2023-01-15 12:38:01 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bfd63a78f93e04350b78ab62bcd335305.dblock.zip.aes (11.03 MB)",
    "2023-01-15 12:38:01 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i46dc54c9de29446597beff39150d687c.dindex.zip.aes (18.61 KB)",
    "2023-01-15 12:38:02 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i46dc54c9de29446597beff39150d687c.dindex.zip.aes (18.61 KB)",
    "2023-01-15 12:38:02 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20230115T202350Z.dlist.zip.aes (12.57 MB)",
    "2023-01-15 12:39:56 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20230115T202350Z.dlist.zip.aes (12.57 MB)",
    "2023-01-15 12:39:56 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed",
    "2023-01-15 12:39:56 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / Keep all, 28.00:00:00 / 7.00:00:00, 181.00:00:00 / 31.00:00:00",
    "2023-01-15 12:39:56 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 1/14/2023 12:00:04 PM, 1/13/2023 12:00:04 PM, 1/12/2023 12:00:04 PM, 1/11/2023 12:00:04 PM, 1/10/2023 12:00:04 PM, 1/9/2023 12:00:04 PM, 1/8/2023 12:00:04 PM, 1/4/2023 12:00:04 PM, 12/28/2022 12:00:04 PM, 12/20/2022 3:00:02 PM, 11/29/2022 12:00:04 PM, 10/24/2022 1:00:04 PM, 9/19/2022 1:00:04 PM, 8/16/2022 1:00:04 PM"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 993,
    "BytesUploaded": 4120731731,
    "BytesDownloaded": 14612072859,
    "FilesUploaded": 159,
    "FilesDownloaded": 279,
    "FilesDeleted": 553,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 68262,
    "KnownFileSize": 1785338349950,
    "LastBackupDate": "2023-01-15T12:23:50-08:00",
    "BackupListCount": 14,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-01-15T20:23:50.094811Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Since then - I had to disable automatic backup. During the last (second attempt on Jan 15) - duplicati downloaded 15 GB worth of data.

This is very surprising as it only put about 130 MB.
Source Files Examined 123338 (1.51 TB) Opened 79 (139.59 MB) Added 71 (130.11 MB) Modified 8 (9.48 MB) Deleted 57

The longest retention for my backup is 6 months. In the past the backup had to delete about 200 GB worth of data but it was never associated with such an excessive amount of download.

Thank you in advance!

2

Any chance you have any advanced options set for this backup (Section “5 Options” of "Configuration: Edit)? I’m no expert, but there are things there that could explain it.

3

Hello

Maybe your backend has aged, and not too well. If it’s fragmented, by default Duplicati will try to compact it. Try to set no-auto-compact to true in the advanced options.

1 Like
4

It is very basic.

image
image871×713 50.8 KB

5

Good to rule that out. Now on to more advanced things like gpatel-fr’s suggestion.

6

wouldn’t it tell me that it is trying to compact it? Or am I misreading / misunderstanding something?

image
image1170×1152 117 KB

This is an interesting thought. However - it somehow does not align with what I am observing.
Please see below. Compacting have run several times since Jan 2. This one is the biggest. 9GB are deleted and the total archive time took 20 min.

image
image1329×564 57.6 KB

I can attach the complete log from Jan 9 however the main point is

"BytesDownloaded": 65615479,
      "FilesUploaded": 3,
      "FilesDownloaded": 3,
      "FilesDeleted": 376,

Duplicati only downloaded 3 files. ~60 MB download

Comparing to today

image
image1176×653 60 KB

Compact phase is not present and we have

    "BytesDownloaded": 14612072859,
      "FilesUploaded": 159,
      "FilesDownloaded": 279,

279 files - 14 GB download. (and it would continue if I didn’t cancel it)

It simply doesn’t add up.

7

I think this phase includes cleaning up after any deletion, including the compact in any of its forms.
There is a lot more information available in About → Show log → Live → Information. An example:

2023-01-09 08:51:54 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-FullyDeletableCount]: Found 0 fully deletable volume(s)
2023-01-09 08:51:54 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-SmallVolumeCount]: Found 2 small volumes(s) with a total size of 8.89 MB
2023-01-09 08:51:54 -05 - [Verbose-Duplicati.Library.Main.Database.LocalDeleteDatabase-WastedSpaceVolumes]: Found 29 volume(s) with a total of 10.55% wasted space (1.28 GB of 12.15 GB)
2023-01-09 08:51:54 -05 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there is 10.55% wasted space and the limit is 10%
...
2023-01-09 08:56:50 -05 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Downloaded 31 file(s) with a total size of 1.25 GB, deleted 62 file(s) with a total size of 1.25 GB, and compacted to 12 file(s) with a size of 264.32 MB, which reduced storage by 50 file(s) and 1,013.13 MB

Above is actually in log-file format. Live log is reverse-chronological, and Information level would only show the highlights. Verbose shows more, and there are levels beyond that, for those who need details.

The cited 10% limit is because I have threshold=10 percent rather than the default 25 to compact often, doing less each time, which is what people do when they’re trying to stay below free limit, i.e. 1 GB/day.

meaning even a small portion of that needing to compact will cause a large download. Only other option besides letting compact run is to not let it run, which means storage used-and-charged grows to the sky. Compromise solution might be to let compact run, but either lower threshold or expect a large download.

Not very relevant. Compact happens not by putting data in, but having existing data age away over time. When a backup version is deleted by your retention policy, any data in no other version becomes waste. Please read these:

Compacting files at the backend
The COMPACT command
no-auto-compact
threshold

This value is a percentage used on each volume and the total storage.

which means sometimes hitting the default 25% triggers a space inspection of your quite large backup, potentially causing a lot of action. The 14 GB that bothers you is less than 1%, and your trigger is 25%. Setting threshold to a very very low value can cause problems, but you might want to lower the setting.

You could, if you like, set it to a larger value, then run a Compact now while watching notices in live log. Testing that, it seemed like Information level wasn’t saying enough, so I went to Verbose and got these:

image
image721×388 24.9 KB

If your wasted space is above 25%, that explains why it’s doing compact. Volume count might also be whatever’s over the threshold on the per-volume view, but I’m less certain, though it seems very likely.

An alternative test would be to set no-auto-compact to see if backup runs, maybe deletes, and ends, however you can’t stay there forever unless you don’t care about running up a really large storage bill.

Verifying backend files
The TEST command

A sample consists of 1 dlist, 1 dindex, 1 dblock.

It’s also in the job log as Test phase, and in Complete log as Test results, naming the actual files.

How are you seeing download amount, and what does the current file reference mean. Is this in live log? Compact has a very recognizable pattern of a bunch of dblock downloads, a dblock upload consolidating downloaded dblock files having too much waste (and I think dindex upload goes with that dblock upload), then a lot of deletes of most of the just recently downloaded dblock files and their associated dindex files.

I haven’t tried cancel in the middle. I’m glad that backup seems to have survived, but stats seem missing. They’re usually result stats done at the end. Compact never got to the end, so maybe that explains zeros.

Here’s a more typical log and Complete log:

image

"CompactResults": {
  "DeletedFileCount": 62,
  "DownloadedFileCount": 31,
  "UploadedFileCount": 12,
  "DeletedFileSize": 1339502534,
  "DownloadedFileSize": 1337742531,
  "UploadedFileSize": 277154502,
  "Dryrun": false,
  "VacuumResults": null,
  "MainOperation": "Compact",
  "ParsedResult": "Success",
  "Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
  "EndTime": "2023-01-09T13:56:51.7248468Z",
  "BeginTime": "2023-01-09T13:51:51.5633202Z",
  "Duration": "00:05:00.1615266",
  "MessagesActualLength": 0,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": null,
  "Warnings": null,
  "Errors": null,
  "BackendStatistics": {
    "RemoteCalls": 116,
    "BytesUploaded": 328322339,
    "BytesDownloaded": 1391097722,
    "FilesUploaded": 16,
    "FilesDownloaded": 34,
    "FilesDeleted": 64,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 111792,
    "UnknownFileCount": 1,
    "KnownFileCount": 469,
    "KnownFileSize": 9171592353,
    "LastBackupDate": "2023-01-09T08:50:01-05:00",
    "BackupListCount": 62,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Version": "2.0.6.104 (2.0.6.104_canary_2022-06-15)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-01-09T13:50:00.0141386Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
},

You can see that compact downloaded 31 files out of 34 total downloads. The other 3 were the Test.

EDIT:

If you decide it’s doing a compact, there’s probably no direct path to doing a lot of smaller compacts because going straight to a low threshold will make an even huger initial one than you’re set to now. Possibly lots of smaller compacts will wind up as a larger download bill, so keep that in mind as well.

EDIT 2:

Compact - Limited / Partial might allow more successful frequent smaller compacts. Currently setting threshold low on the overall backup also sets it low on each volume, so could compact more of them. There are other feature requests for a time limit and a bandwidth limit, and concerns of falling behind.

EDIT 3:

Changing threshold to different higher-than-currently-wasted changes Compact now stats. 10 vs. 95:

Found 19 volume(s) with a total of 6.71% wasted space (815.07 MB of 11.86 GB)

Found 14 volume(s) with a total of 5.47% wasted space (664.72 MB of 11.86 GB)

Multiplying by your remote volume size “might” tell you the amount of downloading you should expect.

EDIT 4:

Sentence above still looks reasonable. Current value is below, and extrapolating to 10% gives 1.2 GB download compared to posted normal actual above of 1.25 GB. Extrapolation is easier for this backup because it generates a fairly fixed amount of new and unique data from file changes each run, and as version retention removes versions, this leads to a fairly steady accumulation of unneeded data waste.

Jan 16, 2023 10:05 AM: Found 23 volume(s) with a total of 8.43% wasted space (1.02 GB of 12.06 GB)

Linear extrapolation: 1.02 GB * 10% / 8.43% = 1.21 GB

Size based on 50 MB remote volumes: 23 * 50 MB = 1.15 GB but seemingly not all of it is waste space.

Download will be larger than wasted space because volumes contain still-in-use data to be compacted.
Volumes that have managed to become 100% wasted space are deleted. There’s nothing useful there.

So there’s how to forecast how much the compact might do, so you can plan when you want to run it…

1 Like
8

Thank you!
I was not aware of this area.

I am going to adjust this setting. It is a good point indeed.

Currently - it appears okay - will provide full summary below

You were right that it is compacting. I was not prepared for it. Did not expect it to do what it did.

So - the summary of what I am getting so far.

My disk is a 3 TB USB plugged to the router EXT4 File System.
I unplugged it and plugged it to the computer - executing fsck -f
It did find a few items to correct.

Once plugged back to the router - duplicati did find discrepancy with the time stamps and it took a while to correct that (several hours).

I left the computer overnight allowing bigger quota.
At the end the backup completed and it had to download 79 GB.

Full log below

{
  "DeletedFiles": 0,
  "DeletedFolders": 0,
  "ModifiedFiles": 2,
  "ExaminedFiles": 123345,
  "OpenedFiles": 21797,
  "AddedFiles": 21795,
  "SizeOfModifiedFiles": 10598180,
  "SizeOfAddedFiles": 844579852600,
  "SizeOfExaminedFiles": 1661732109361,
  "SizeOfOpenedFiles": 844590450780,
  "NotProcessedFiles": 0,
  "AddedFolders": 1063,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 0,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 3062,
    "DownloadedFileCount": 1531,
    "UploadedFileCount": 856,
    "DeletedFileSize": 78744876686,
    "DownloadedFileSize": 78682819231,
    "UploadedFileSize": 22431942884,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-01-16T13:09:31.054889Z",
    "BeginTime": "2023-01-16T09:29:19.797419Z",
    "Duration": "03:40:11.2574700",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5458,
      "BytesUploaded": 22456429935,
      "BytesDownloaded": 78746061654,
      "FilesUploaded": 859,
      "FilesDownloaded": 1534,
      "FilesDeleted": 3062,
      "FoldersCreated": 0,
      "RetryAttempts": 1,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 66060,
      "KnownFileSize": 1729014044348,
      "LastBackupDate": "2023-01-15T18:41:53-08:00",
      "BackupListCount": 16,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-16T02:41:49.336821Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 0,
    "DeletedSets": [],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": {
      "DeletedFileCount": 3062,
      "DownloadedFileCount": 1531,
      "UploadedFileCount": 856,
      "DeletedFileSize": 78744876686,
      "DownloadedFileSize": 78682819231,
      "UploadedFileSize": 22431942884,
      "Dryrun": false,
      "VacuumResults": null,
      "MainOperation": "Compact",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "2023-01-16T13:09:31.054889Z",
      "BeginTime": "2023-01-16T09:29:19.797419Z",
      "Duration": "03:40:11.2574700",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null,
      "BackendStatistics": {
        "RemoteCalls": 5458,
        "BytesUploaded": 22456429935,
        "BytesDownloaded": 78746061654,
        "FilesUploaded": 859,
        "FilesDownloaded": 1534,
        "FilesDeleted": 3062,
        "FoldersCreated": 0,
        "RetryAttempts": 1,
        "UnknownFileSize": 0,
        "UnknownFileCount": 0,
        "KnownFileCount": 66060,
        "KnownFileSize": 1729014044348,
        "LastBackupDate": "2023-01-15T18:41:53-08:00",
        "BackupListCount": 16,
        "TotalQuotaSpace": 0,
        "FreeQuotaSpace": 0,
        "AssignedQuotaSpace": -1,
        "ReportedQuotaError": false,
        "ReportedQuotaWarning": false,
        "MainOperation": "Backup",
        "ParsedResult": "Success",
        "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
        "EndTime": "0001-01-01T00:00:00",
        "BeginTime": "2023-01-16T02:41:49.336821Z",
        "Duration": "00:00:00",
        "MessagesActualLength": 0,
        "WarningsActualLength": 0,
        "ErrorsActualLength": 0,
        "Messages": null,
        "Warnings": null,
        "Errors": null
      }
    },
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-01-16T13:09:31.054961Z",
    "BeginTime": "2023-01-16T09:28:03.410602Z",
    "Duration": "03:41:27.6443590",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5458,
      "BytesUploaded": 22456429935,
      "BytesDownloaded": 78746061654,
      "FilesUploaded": 859,
      "FilesDownloaded": 1534,
      "FilesDeleted": 3062,
      "FoldersCreated": 0,
      "RetryAttempts": 1,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 66060,
      "KnownFileSize": 1729014044348,
      "LastBackupDate": "2023-01-15T18:41:53-08:00",
      "BackupListCount": 16,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-16T02:41:49.336821Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20230116T011735Z.dlist.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-i5a0414fffaf549bfb97812c25dee82a3.dindex.zip.aes",
        "Value": []
      },
      {
        "Key": "duplicati-b7a9f8806a1834031833172bde392709a.dblock.zip.aes",
        "Value": []
      }
    ],
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-01-16T13:14:13.852659Z",
    "BeginTime": "2023-01-16T13:10:30.051553Z",
    "Duration": "00:03:43.8011060",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 5458,
      "BytesUploaded": 22456429935,
      "BytesDownloaded": 78746061654,
      "FilesUploaded": 859,
      "FilesDownloaded": 1534,
      "FilesDeleted": 3062,
      "FoldersCreated": 0,
      "RetryAttempts": 1,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 66060,
      "KnownFileSize": 1729014044348,
      "LastBackupDate": "2023-01-15T18:41:53-08:00",
      "BackupListCount": 16,
      "TotalQuotaSpace": 0,
      "FreeQuotaSpace": 0,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-01-16T02:41:49.336821Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Success",
  "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
  "EndTime": "2023-01-16T13:14:13.863521Z",
  "BeginTime": "2023-01-16T02:41:49.336815Z",
  "Duration": "10:32:24.5267060",
  "MessagesActualLength": 10927,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 0,
  "Messages": [
    "2023-01-15 18:41:49 -08 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
    "2023-01-15 18:42:39 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2023-01-15 18:43:28 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (66.66 KB)",
    "2023-01-15 18:43:32 -08 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-bd6f12f2a9d2f4e2fabf6c12d00904176.dblock.zip.aes",
    "2023-01-15 18:43:32 -08 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-i6a38af6fe4a646fe8203d0f7cd6a0291.dindex.zip.aes",
    "2023-01-16 01:26:25 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b4ff5efc19fc7444caf1238e0818be616.dblock.zip.aes (508.28 KB)",
    "2023-01-16 01:27:10 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b4ff5efc19fc7444caf1238e0818be616.dblock.zip.aes (508.28 KB)",
    "2023-01-16 01:27:10 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i46c16b7c57ae4569bb1db23edc83d2d5.dindex.zip.aes (5.92 KB)",
    "2023-01-16 01:27:10 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i46c16b7c57ae4569bb1db23edc83d2d5.dindex.zip.aes (5.92 KB)",
    "2023-01-16 01:27:11 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-20230116T024153Z.dlist.zip.aes (12.57 MB)",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-20230116T024153Z.dlist.zip.aes (12.57 MB)",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-StartCheck]: Start checking if backups can be removed",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-FramesAndIntervals]: Time frames and intervals pairs: 7.00:00:00 / Keep all, 28.00:00:00 / 7.00:00:00, 181.00:00:00 / 31.00:00:00",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupList]: Backups to consider: 1/15/2023 5:17:35 PM, 1/15/2023 12:23:50 PM, 1/14/2023 12:00:04 PM, 1/13/2023 12:00:04 PM, 1/12/2023 12:00:04 PM, 1/11/2023 12:00:04 PM, 1/10/2023 12:00:04 PM, 1/9/2023 12:00:04 PM, 1/4/2023 12:00:04 PM, 12/28/2022 12:00:04 PM, 12/20/2022 3:00:02 PM, 11/29/2022 12:00:04 PM, 10/24/2022 1:00:04 PM, 9/19/2022 1:00:04 PM, 8/16/2022 1:00:04 PM",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-BackupsToDelete]: Backups outside of all time frames and thus getting deleted: ",
    "2023-01-16 01:28:03 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler:RetentionPolicy-AllBackupsToDelete]: All backups to delete: ",
    "2023-01-16 01:29:19 -08 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted",
    "2023-01-16 01:32:18 -08 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 60.80 MB in small volumes and the volume size is 50.00 MB",
    "2023-01-16 01:36:39 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bd9378a2749a84ce2863ebadfe9d41152.dblock.zip.aes (49.93 MB)",
    "2023-01-16 01:36:42 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bd9378a2749a84ce2863ebadfe9d41152.dblock.zip.aes (49.93 MB)"
  ],
  "Warnings": [],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 5458,
    "BytesUploaded": 22456429935,
    "BytesDownloaded": 78746061654,
    "FilesUploaded": 859,
    "FilesDownloaded": 1534,
    "FilesDeleted": 3062,
    "FoldersCreated": 0,
    "RetryAttempts": 1,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 66060,
    "KnownFileSize": 1729014044348,
    "LastBackupDate": "2023-01-15T18:41:53-08:00",
    "BackupListCount": 16,
    "TotalQuotaSpace": 0,
    "FreeQuotaSpace": 0,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-01-16T02:41:49.336821Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

I am planning to apply the advanced option for threshold as per @ts678 and will have to monitor how it moves forward.

Thank you all!
I am going to observe for few days and if all works well will mark this resolved. Of course if I stumble again will have to ask you again.

Appreciate all the help!

9

and it found about 56 GB still in use, which means about 28.5% was waste, which I guess makes sense because some volumes might have had more waste. If anybody ever adds a dual-threshold feature, one possible way to use it would be to compact volumes that have quite a bit of waste, but set threshold low across the whole backup so compact cleans them up fewer at a time as their contents turn into waste…

In B2 pricing terms, the 79 cent download is saving around 28 cents a month by reducing wasted space. Working out what works best may take some trial and error. Being too sensitive to space waste might do very little actual waste removal per download/repackage/upload cycle, and that drives up download fees.

Good luck, but now you know things about compact. If it helps any, you can schedule it in Task Manager.

1 Like
10

I’ve noticed one big difference between Restic and Duplicati. This is based on feeling and observation, not source / facts.

  1. Restic compacts individual files when any file exceeds the deleted data threshold.
  2. Duplicati compacts while backup, when the backup exceeds the deleted data threshold.

This means that Restic compacts are much more spread out and of course there’s also more churn. Afaik, it would be nice to have option to choose from these. I think it would be useful in my use case, because it would also limit the time individual compaction runs take.

Could there be an option for that?

And of course this doesn’t mean that the transactionality issue wouldn’t need fixing. But it would make it less likely to get “hit”.

11

Are you certain about that? After seeing the results of a threshold change from my EDIT 3,
it’s looking like maybe it looks at volumes with too much wasted data then adds up wastes.
Comparison to threshold on total backup then follows, but (sadly) thresholds are the same.

https://duplicati.com/assets/Block-basedstorageformat.pdf which was cited yesterday says:

The initial version is to calculate the size of blocks that cannot be reached in each volume, and
once a certain threshold is exceeded, the volume is marked as wasteful. As soon as the overall
waste exceeds the size of a volume, wasteful volume(s) are downloaded and new volumes are
written.

Aside from “overall waste exceeds the size of a volume” (fixed size replaced by threshold option?), paragraph processing order fits my finding that raising threshold reduces quantity of volumes listed (“marked as wasteful” concept above), and of course summing fewer volumes reduces total waste.

looks to me (who doesn’t do much C#) like what I described, but it’s just the report, not the compact.
Assuming all is well, the key point is that it’s not looking at the total waste, just the waste in volumes individually exceeding the waste threshold, so if we can ever get the dual-threshold you proposed, a possible solution would be to set per-volume threshold at some generous amount, set total threshold based on how often we’d hope to compact. Compacting requires several volumes to be available, as created new volumes get filled, and by definition a volume with waste won’t fully fill one all on its own.

Does it just upload a small file then? You can see that most of mine were 5% full or less, and very little would be uploaded back. This possibly unusual pattern would better suit Duplicati’s pack-it-full method. Even restic seems like it could have a flurry of compacts if a lot of volumes hit waste limit all at once…

I think the dual threshold might be easier, and maybe enough to have more but shorter compact runs. Regardless, nothing (in any code area) happens without developers, and any interested one can look.

12

To keep it short. Yes and yes. Restic generates many more more smalls than duplicati does. And actually that’s one of the things I like with Duplicati and SMR backup disks.

Also reducing compact interval, generates more churn. Because now more files are downloaded and rewritten to back-end while less data gets deleted and therefore more data is uploaded back.

As stated, there’s no perfect solution, just different trade-offs. With my use case with Restic, it compacts something almost on every run.