Error on backup!

Support
1

Good morning!
Recently I am having this error when completing the backup routine.
I already repaired the database, verified files and still with the same problem!
Please, I need help solving this problem, I don’t even know where to start!

Error:
2023-02-26 10:41:34 -03 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b6d2e94e149e64cb99c1a29dea3cbe57e.dblock.zip

Complete Log:

{
  "DeletedFiles": 0,
  "DeletedFolders": 0,
  "ModifiedFiles": 0,
  "ExaminedFiles": 726252,
  "OpenedFiles": 0,
  "AddedFiles": 0,
  "SizeOfModifiedFiles": 0,
  "SizeOfAddedFiles": 0,
  "SizeOfExaminedFiles": 809180276857,
  "SizeOfOpenedFiles": 0,
  "NotProcessedFiles": 0,
  "AddedFolders": 0,
  "TooLargeFiles": 0,
  "FilesWithError": 0,
  "ModifiedFolders": 0,
  "ModifiedSymlinks": 0,
  "AddedSymlinks": 0,
  "DeletedSymlinks": 0,
  "PartialBackup": false,
  "Dryrun": false,
  "MainOperation": "Backup",
  "CompactResults": {
    "DeletedFileCount": 2,
    "DownloadedFileCount": 0,
    "UploadedFileCount": 0,
    "DeletedFileSize": 1080,
    "DownloadedFileSize": 0,
    "UploadedFileSize": 0,
    "Dryrun": false,
    "VacuumResults": null,
    "MainOperation": "Compact",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-02-26T06:12:32.3648578Z",
    "BeginTime": "2023-02-26T06:09:29.1836118Z",
    "Duration": "00:03:03.1812460",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 1080,
      "BytesDownloaded": 15891888320,
      "FilesUploaded": 2,
      "FilesDownloaded": 7,
      "FilesDeleted": 2,
      "FoldersCreated": 0,
      "RetryAttempts": 4,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 85,
      "KnownFileSize": 292961549065,
      "LastBackupDate": "2023-02-25T02:00:00-03:00",
      "BackupListCount": 7,
      "TotalQuotaSpace": 11380623242478,
      "FreeQuotaSpace": 10995116277760,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-02-26T05:00:00.3589511Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "VacuumResults": null,
  "DeleteResults": {
    "DeletedSetsActualLength": 0,
    "DeletedSets": [],
    "Dryrun": false,
    "MainOperation": "Delete",
    "CompactResults": {
      "DeletedFileCount": 2,
      "DownloadedFileCount": 0,
      "UploadedFileCount": 0,
      "DeletedFileSize": 1080,
      "DownloadedFileSize": 0,
      "UploadedFileSize": 0,
      "Dryrun": false,
      "VacuumResults": null,
      "MainOperation": "Compact",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "2023-02-26T06:12:32.3648578Z",
      "BeginTime": "2023-02-26T06:09:29.1836118Z",
      "Duration": "00:03:03.1812460",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null,
      "BackendStatistics": {
        "RemoteCalls": 13,
        "BytesUploaded": 1080,
        "BytesDownloaded": 15891888320,
        "FilesUploaded": 2,
        "FilesDownloaded": 7,
        "FilesDeleted": 2,
        "FoldersCreated": 0,
        "RetryAttempts": 4,
        "UnknownFileSize": 0,
        "UnknownFileCount": 0,
        "KnownFileCount": 85,
        "KnownFileSize": 292961549065,
        "LastBackupDate": "2023-02-25T02:00:00-03:00",
        "BackupListCount": 7,
        "TotalQuotaSpace": 11380623242478,
        "FreeQuotaSpace": 10995116277760,
        "AssignedQuotaSpace": -1,
        "ReportedQuotaError": false,
        "ReportedQuotaWarning": false,
        "MainOperation": "Backup",
        "ParsedResult": "Success",
        "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
        "EndTime": "0001-01-01T00:00:00",
        "BeginTime": "2023-02-26T05:00:00.3589511Z",
        "Duration": "00:00:00",
        "MessagesActualLength": 0,
        "WarningsActualLength": 0,
        "ErrorsActualLength": 0,
        "Messages": null,
        "Warnings": null,
        "Errors": null
      }
    },
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-02-26T06:12:32.3648578Z",
    "BeginTime": "2023-02-26T06:07:53.240521Z",
    "Duration": "00:04:39.1243368",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 1080,
      "BytesDownloaded": 15891888320,
      "FilesUploaded": 2,
      "FilesDownloaded": 7,
      "FilesDeleted": 2,
      "FoldersCreated": 0,
      "RetryAttempts": 4,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 85,
      "KnownFileSize": 292961549065,
      "LastBackupDate": "2023-02-25T02:00:00-03:00",
      "BackupListCount": 7,
      "TotalQuotaSpace": 11380623242478,
      "FreeQuotaSpace": 10995116277760,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-02-26T05:00:00.3589511Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "RepairResults": null,
  "TestResults": {
    "MainOperation": "Test",
    "VerificationsActualLength": 3,
    "Verifications": [
      {
        "Key": "duplicati-20230215T050000Z.dlist.zip",
        "Value": []
      },
      {
        "Key": "duplicati-ie86bcd9892f84f7c8b1889efe3f2937b.dindex.zip",
        "Value": []
      },
      {
        "Key": "duplicati-b37541b142d5546868ee11705ea7fad66.dblock.zip",
        "Value": [
          {
            "Key": "Error",
            "Value": "O arquivo duplicati-b37541b142d5546868ee11705ea7fad66.dblock.zip foi baixado e possui tamanho 3607101440, porém o tamanho esperado deveria ser 10742077800"
          }
        ]
      }
    ],
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "2023-02-26T06:23:54.2878304Z",
    "BeginTime": "2023-02-26T06:12:40.1618113Z",
    "Duration": "00:11:14.1260191",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null,
    "BackendStatistics": {
      "RemoteCalls": 13,
      "BytesUploaded": 1080,
      "BytesDownloaded": 15891888320,
      "FilesUploaded": 2,
      "FilesDownloaded": 7,
      "FilesDeleted": 2,
      "FoldersCreated": 0,
      "RetryAttempts": 4,
      "UnknownFileSize": 0,
      "UnknownFileCount": 0,
      "KnownFileCount": 85,
      "KnownFileSize": 292961549065,
      "LastBackupDate": "2023-02-25T02:00:00-03:00",
      "BackupListCount": 7,
      "TotalQuotaSpace": 11380623242478,
      "FreeQuotaSpace": 10995116277760,
      "AssignedQuotaSpace": -1,
      "ReportedQuotaError": false,
      "ReportedQuotaWarning": false,
      "MainOperation": "Backup",
      "ParsedResult": "Success",
      "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
      "EndTime": "0001-01-01T00:00:00",
      "BeginTime": "2023-02-26T05:00:00.3589511Z",
      "Duration": "00:00:00",
      "MessagesActualLength": 0,
      "WarningsActualLength": 0,
      "ErrorsActualLength": 0,
      "Messages": null,
      "Warnings": null,
      "Errors": null
    }
  },
  "ParsedResult": "Error",
  "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
  "EndTime": "2023-02-26T06:23:55.3503396Z",
  "BeginTime": "2023-02-26T05:00:00.3589511Z",
  "Duration": "01:23:54.9913885",
  "MessagesActualLength": 37,
  "WarningsActualLength": 0,
  "ErrorsActualLength": 1,
  "Messages": [
    "2023-02-26 02:00:01 -03 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: A operação Backup foi iniciada",
    "2023-02-26 02:13:20 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2023-02-26 02:13:24 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (85 bytes)",
    "2023-02-26 02:49:37 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b22f96aa16e2f4444aa92808eca548ce3.dblock.zip (514 bytes)",
    "2023-02-26 02:50:35 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b22f96aa16e2f4444aa92808eca548ce3.dblock.zip (514 bytes)",
    "2023-02-26 03:07:50 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i6de313cf43bb44cebf04b25205efd83f.dindex.zip (566 bytes)",
    "2023-02-26 03:07:53 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i6de313cf43bb44cebf04b25205efd83f.dindex.zip (566 bytes)",
    "2023-02-26 03:09:29 -03 - [Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]: No remote filesets were deleted",
    "2023-02-26 03:12:24 -03 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting because there are 1 fully deletable volume(s)",
    "2023-02-26 03:12:25 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-b22f96aa16e2f4444aa92808eca548ce3.dblock.zip (514 bytes)",
    "2023-02-26 03:12:26 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-b22f96aa16e2f4444aa92808eca548ce3.dblock.zip (514 bytes)",
    "2023-02-26 03:12:26 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Started: duplicati-i6de313cf43bb44cebf04b25205efd83f.dindex.zip (566 bytes)",
    "2023-02-26 03:12:26 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Delete - Completed: duplicati-i6de313cf43bb44cebf04b25205efd83f.dindex.zip (566 bytes)",
    "2023-02-26 03:12:32 -03 - [Information-Duplicati.Library.Main.Operation.CompactHandler-CompactResults]: Deleted 2 files, which reduced storage by 1,05 KB",
    "2023-02-26 03:12:32 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2023-02-26 03:12:34 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (85 bytes)",
    "2023-02-26 03:12:34 -03 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-bee00d20ef5894886907cc118aa12c946.dblock.zip",
    "2023-02-26 03:12:34 -03 - [Information-Duplicati.Library.Main.Operation.FilelistProcessor-RemoteUnwantedMissingFile]: removing file listed as Temporary: duplicati-i4119bd65345b494da775b31e4cba442f.dindex.zip",
    "2023-02-26 03:12:40 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20230215T050000Z.dlist.zip (78,40 MB)",
    "2023-02-26 03:12:59 -03 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20230215T050000Z.dlist.zip (78,40 MB)"
  ],
  "Warnings": [],
  "Errors": [
    "2023-02-26 03:23:54 -03 - [Error-Duplicati.Library.Main.Operation.TestHandler-FailedToProcessFile]: Failed to process file duplicati-b37541b142d5546868ee11705ea7fad66.dblock.zip"
  ],
  "BackendStatistics": {
    "RemoteCalls": 13,
    "BytesUploaded": 1080,
    "BytesDownloaded": 15891888320,
    "FilesUploaded": 2,
    "FilesDownloaded": 7,
    "FilesDeleted": 2,
    "FoldersCreated": 0,
    "RetryAttempts": 4,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 85,
    "KnownFileSize": 292961549065,
    "LastBackupDate": "2023-02-25T02:00:00-03:00",
    "BackupListCount": 7,
    "TotalQuotaSpace": 11380623242478,
    "FreeQuotaSpace": 10995116277760,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Backup",
    "ParsedResult": "Success",
    "Version": "2.0.6.3 (2.0.6.3_beta_2021-06-17)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2023-02-26T05:00:00.3589511Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}
2

Welcome to the forum @RobsonZornitta

Verifying backend files

At the end of each backup job, Duplicati checks the integrity by downloading a few files from the backend. The contents of these file is checked against what Duplicati expects it to be.

Translation of the error value from the right hand side of last line:

was downloaded and has size 3607101440, but the expected size should be 10742077800

which means in rough numbers that a 10 GB file turned into a 3 GB file. What storage is it on?

Can you post file date and size from some other tool? Can you open file with a .zip program?

If file is truly gone, that’s a pretty big loss compared to the default 50 MB shown on this screen:

image
image1016×277 16.9 KB

Above is an image, but “See this page” on Duplicati probably takes you to Remote Volume Size
which tries to explain it further. Some people mistakenly set it to the total capacity of destination.

EDIT:

3607101440 is a suspiciously even D7000000 in hexadecimal, suggesting a filesystem problem,
although some other things tend to work in binary units. SMB share might also be able to do this.
When inspecting the file, try to get as close to it as possible, e.g. if it’s remote, look from that end.

3

Good morning!
Thanks a lot for answering!
The cloud storage is on Google Drive, right now, I’m running the backup task, I recreated the database and ran the backup task, I intend to recreate the task because the current backup is too big. What size compressed files would be recommended for uploading?

4

You’re very welcome, but can you do the file tests in the answer – at least give the file size and date.

Ultimately knowing if/how damage happened would be nice so maybe ways can be found to prevent.
Merely setting an overly large remote volume size should not by itself cause Google to kill file AFAIK.

Downloading the file could be a little slow depending on your speed – and that also affects best size, because restoring even a single file may need to download all the dblock files with blocks of file data.

This is explained in the linked article. Please read that or at least say something about your situation.

EDIT:

Internet search finds claims that Google Drive has a 5 TB size limit, however in practice (and again it depends on the speed and quality of your network), large file transfers might have trouble completing, which is why I keep asking what the size of the file seems to be at file storage location (not your end).

There was also useful information in logs and tables in the old database, but recreate has lost history.

EDIT 2:

Testing with a small .zip file (possibly large ones will fail due to .zip limitation), Google Drives’s web viewer at drive.google.com can open them for your test, and can also display file size and date info, however the ending of a backup does try to check its files from a directory listing before test download.

image

Above is reverse-chronological order, and you can compare to your post. You got dlist and dindex, however something went wrong with dblock, so it didn’t all get down after number-of-retries finished.

If you still had your old database, the Complete log of the job log would show its "RetryAttempts" which you can keep an eye on in future backups to try to guess whether your Internet is giving errors.

5

I apologize for not reporting such data!
The files are not encrypted and have a file size of 10GB, the backup started having problems 10 days ago.
The backup started to be done on October 9, 2022, without major problems, until now!

6

I guess you’re giving the Remote volume size, but I’d like information on a specific failing dblock:

duplicati-b37541b142d5546868ee11705ea7fad66.dblock.zip

This should be easy to read off the web page, and clicking the file should be an easy test for unzip.

7

Sorry!

image
image1366×768 90.7 KB

I’m downloading it to do the unzipping test!

8

OK, so it was a partial download that exceeded the retry limit. As noted later, you can probably just click that line on your screen to look inside it courtesty of Google (if they can stand the size), but download is probably a good way to see if it takes annoyingly long, and whether it makes it. Might have been a brief Internet or LAN glitch too. Is it always this file (if you’ve noticed) or might it be a random occasional file?

EDIT:

If the DB recreate succeeded without complaining about any files (did it?) then backup downloaded OK.
Actually that’s not correct, as it may have just done dlist and dindex files. Were you watching closely
enough to see how far it had to go on the progress bar? If past 70%, especially past 90%, it got dblocks.

1 Like
9

I think I found the problem, I just can’t download this file, I’ve tried 6 times and the download fails every time!

Unfortunately I didn’t personally follow the completion of the backup but now I can analyze that the problem is in Google Drive, I already checked the files on the source HD I recreated the Duplicati DB, but I didn’t know it wasn’t legal on Google!
I’ll recreate the backup with smaller files to see if it works!

10

It might be that your LAN or Internet connection (or the Internet itself) is too unreliable to move big files.
A seemingly far-fetched idea is that Google lost that one, and some other big one would download fine.
Even if you have a big maximum size, you’re not going to get big files unless there’s that much needed.

Another thing to do is to increase blocksize some, maybe to 500 KB, or 1 MB if backup may grow a lot. Tracking more than a million blocks can slow things down, so default 100 KB is good for about 100 GB.

You can’t change blocksize except on a new backup. Remote volume size can be raised (which will do compact, so maybe raise it slowly if you raise it). I don’t think it gains much to go huge on the volumes, however typical Internet speeds have been going up (I don’t know yours), so some raising may be fine.

1 Like
11

Kind of what I was thinking. Though Google Drive can get really slow at times. Don’t think I’ve ever seen it actually drop a file. If a file is being sent then a connection issue or Drive limit is far more likely. And it looks like it doesn’t have a limit here at these sizes today.

@RobsonZornitta Fixing your internet connection quality if it has a problem here is a top of the list thing to do. Good working network and internet makes things more pleasant.

Google actually does check files, as if its a movie or something, they don’t want them on there. But I don’t think that’s the issue here unless they made the scan time fast (not expected). Just try making a large 5GB zip of junk and see if it works better and also try that directly through Drive’s UI if you have the bandwidth as if it works fine outside of Duplicati then there’s something :slight_smile:

12

I’m going to open a ticket on Google for the file in question, I have Workspace paid for 500 Standard users, I tried to download it from 4 different places with different internet providers and really, the file does not complete the download, failures occur every time network!
I don’t know if it’s common for this to happen on Google Drive, but I’ll monitor this situation better. As soon as I manage to resolve this situation, I will post the solution! Thanks a lot for the help!

13

If you think Google chopped it, you could try opening that .zip file in their web UI to see what happens.
You can try copying it in their web UI to see what happens. All this might slow down blaming networks.
I don’t know if it will convince them, but that super-binary-even size still sounds like it might be storage.

If you can get it to happen again with Duplicati, you can see if it always fails at that size (watch live log About → Show log → Live → Retry), that would be an even stronger indication that it’s not the network.

If it’s just one file (any idea?) you could instead do a command line download, i.e. Export As Command-line and use URL in Duplicati.CommandLine.BackendTool.exe get. Maybe try a smaller file for practice.

14

Good afternoon!
In fact, I deleted the entire backup, both in Duplicati and Google, I’m going to upload another backup, now with smaller and more segmented files to test! As soon as I have a review I’ll post it here!
Thanks in advance to everyone who helped!