Hi,
Today Duplicati has run. No new file had been added to my computer since the last update.
However I have noticed it has created one .dindex and one .dblock file (no .dlist file) on the backend
The .dblock file contains two small chuncks that are not linked to a file in the database and that contain this information:
{“win-ext:accessrules”:"",“CoreAttributes”:“ReadOnly, Directory”,“CoreLastWritetime”:“637520990410819095”,“CoreCreatetime”:“636724582029628046”}
What is it for? Is it really needed? I would like to limit the number of files on the back-end, is there an option to avoid having those files added?
Thanks
That looks like times for file created GMT / UTC Date Time: 2018-09-13 17:56:42 and modified GMT / UTC Date Time: 2021-03-23 12:24:01, which may qualify it as not a new file, per your post. What OS is this on? Different ones use the two timestamp fields differently. Lack of dlist is surprising, but maybe there’s a way.
You mentioned another file within the dblock .zip file, but I don’t see it. Possibly it’s data changes to the file.
The AFFECTED command is what to run to check file associations (or not). You can run in Commandline, converting command and syntax as necessary. The file (not path) names of the files in question are put in.
The COMPARE command can also be used to see what might have changed, e.g. if the file was modified.
Viewing the log files of a backup job is probably the easiest way. It gives a count of modified files. Example:
What are you using that’s sensitive to number? There are lots of ways to help that (with some tradeoffs).
Choosing sizes in Duplicati
Thanks for you answer.
When I run the Affected command, the answer is :
total of 2 file(s) are affected:
C:\Users\zorro\Desktop\Google drive
D:\Archives\Documents perso\
Those are folders. It explains why there is no .dlist file. But how is it possible that some folders are considered as modified (and uploaded) whereas they are just folders and that no file inside them have changed?
The log shows:
Examined 194524 Opened:0 Added: 0 Modified: 0 Deleted: 1
The log is
{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 194524,
“OpenedFiles”: 0,
“AddedFiles”: 0,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 0,
“SizeOfExaminedFiles”: 238081873045,
“SizeOfOpenedFiles”: 0,
“NotProcessedFiles”: 0,
“AddedFolders”: 0,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: null,
“VacuumResults”: null,
“DeleteResults”: null,
“RepairResults”: null,
“TestResults”: null,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “2021-03-23T12:32:41.6344772Z”,
“BeginTime”: “2021-03-23T12:24:09.3064144Z”,
“Duration”: “00:08:32.3280628”,
“MessagesActualLength”: 10,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2021-03-23 13:24:12 +01 - [Information-GetGpgProgramPath-gpg]: C:\ProgramData\Duplicati\updates\2.0.5.1\win-tools\gpg.exe”,
“2021-03-23 13:24:12 +01 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: L’opération Backup a débuté”,
“2021-03-23 13:25:35 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2021-03-23 13:25:44 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (1.65 KB)”,
“2021-03-23 13:32:19 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-bdf15dd172f0c4a91b7328df87b6b86aa.dblock.zip.aes (1.09 KB)”,
“2021-03-23 13:32:32 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-bdf15dd172f0c4a91b7328df87b6b86aa.dblock.zip.aes (1.09 KB)”,
“2021-03-23 13:32:33 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-ie283e45ccc6b46289021eae6938896db.dindex.zip.aes (925 octets)”,
“2021-03-23 13:32:35 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-ie283e45ccc6b46289021eae6938896db.dindex.zip.aes (925 octets)”,
“2021-03-23 13:32:36 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()”,
“2021-03-23 13:32:41 +01 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (1.66 KB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 4,
“BytesUploaded”: 2042,
“BytesDownloaded”: 0,
“FilesUploaded”: 2,
“FilesDownloaded”: 0,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 1696,
“KnownFileSize”: 221301954992,
“LastBackupDate”: “2021-03-20T13:00:00+01:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.1 (2.0.5.1_beta_2020-01-18)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2021-03-23T12:24:09.314412Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}
What was below? If it was in a folder, deleting it means containing folder had Date modified updated.
If you don’t know what above was, a compare might help. As for Google Drive, who knows what it does.
Regardless, you can check folder times with Windows File Explorer, do you your own experiments, etc.
Two things about your logs look odd, and might call for compare. If that’s the same run for the summary and the Complete log, summary shows 1 Deleted file and Complete log shows 0. Also, I think dlist has modification times, so should be updated, but I think there’s a bug that only content change does so:
Sorry it is my mistake, I copied by hand. the real log is:
Examined 194524
Opened:0
Added: 0
Modified: 0
Deleted: 0
So no file changed but the folder them self are considered changed and uploaded. How is it possible?
Thanks
Possibly some looseness in the statistics. I just ran a test where I backed up a folder. I then added and deleted a new file in the folder. The result is no changed files, although timestamp on folder got update. Running a backup got a dblock and dindex file uploaded, just like yours. Inside the dblock are metadata
{“win-ext:accessrules”:“”,“CoreAttributes”:“Directory”,“CoreLastWritetime”:“637524730655652034”,“CoreCreatetime”:“637521152431632186”}
You can see these statistics don’t show the modified timestamp on folder, but the title does say Files.
Complete log is similar to yours. I don’t know why ModifiedFolders is 0, but I don’t know what it counts:
{
"DeletedFiles": 0,
"DeletedFolders": 0,
"ModifiedFiles": 0,
"ExaminedFiles": 1,
"OpenedFiles": 0,
"AddedFiles": 0,
"SizeOfModifiedFiles": 0,
"SizeOfAddedFiles": 0,
"SizeOfExaminedFiles": 0,
"SizeOfOpenedFiles": 0,
"NotProcessedFiles": 0,
"AddedFolders": 0,
"TooLargeFiles": 0,
"FilesWithError": 0,
"ModifiedFolders": 0,
"ModifiedSymlinks": 0,
"AddedSymlinks": 0,
"DeletedSymlinks": 0,
"PartialBackup": false,
"Dryrun": false,
"MainOperation": "Backup",
"CompactResults": {
"DeletedFileCount": 0,
"DownloadedFileCount": 0,
"UploadedFileCount": 0,
"DeletedFileSize": 0,
"DownloadedFileSize": 0,
"UploadedFileSize": 0,
"Dryrun": false,
"VacuumResults": null,
"MainOperation": "Compact",
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "2021-03-27T20:17:53.5233595Z",
"BeginTime": "2021-03-27T20:17:53.5214045Z",
"Duration": "00:00:00.0019550",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null,
"BackendStatistics": {
"RemoteCalls": 7,
"BytesUploaded": 1080,
"BytesDownloaded": 1802,
"FilesUploaded": 2,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 5,
"KnownFileSize": 3183,
"LastBackupDate": "2021-03-27T16:17:04-04:00",
"BackupListCount": 1,
"TotalQuotaSpace": 999618043904,
"FreeQuotaSpace": 467196026880,
"AssignedQuotaSpace": -1,
"ReportedQuotaError": false,
"ReportedQuotaWarning": false,
"MainOperation": "Backup",
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2021-03-27T20:17:52.9639877Z",
"Duration": "00:00:00",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null
}
},
"VacuumResults": null,
"DeleteResults": null,
"RepairResults": null,
"TestResults": {
"MainOperation": "Test",
"VerificationsActualLength": 3,
"Verifications": [
{
"Key": "duplicati-20210327T201704Z.dlist.zip",
"Value": []
},
{
"Key": "duplicati-i9a89c2f06366401291b4c2b10355c6d2.dindex.zip",
"Value": []
},
{
"Key": "duplicati-b751aedacbbf94d56a23ced2ca7b24795.dblock.zip",
"Value": []
}
],
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "2021-03-27T20:17:54.0913915Z",
"BeginTime": "2021-03-27T20:17:53.8304652Z",
"Duration": "00:00:00.2609263",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null,
"BackendStatistics": {
"RemoteCalls": 7,
"BytesUploaded": 1080,
"BytesDownloaded": 1802,
"FilesUploaded": 2,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 5,
"KnownFileSize": 3183,
"LastBackupDate": "2021-03-27T16:17:04-04:00",
"BackupListCount": 1,
"TotalQuotaSpace": 999618043904,
"FreeQuotaSpace": 467196026880,
"AssignedQuotaSpace": -1,
"ReportedQuotaError": false,
"ReportedQuotaWarning": false,
"MainOperation": "Backup",
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2021-03-27T20:17:52.9639877Z",
"Duration": "00:00:00",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null
}
},
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "2021-03-27T20:17:54.3712505Z",
"BeginTime": "2021-03-27T20:17:52.9639877Z",
"Duration": "00:00:01.4072628",
"MessagesActualLength": 16,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": [
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Backup has started",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (3 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-b751aedacbbf94d56a23ced2ca7b24795.dblock.zip (512 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b751aedacbbf94d56a23ced2ca7b24795.dblock.zip (512 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Started: duplicati-i9a89c2f06366401291b4c2b10355c6d2.dindex.zip (568 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-i9a89c2f06366401291b4c2b10355c6d2.dindex.zip (568 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]: Compacting not required",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started: ()",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed: (5 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-20210327T201704Z.dlist.zip (722 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-20210327T201704Z.dlist.zip (722 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-i9a89c2f06366401291b4c2b10355c6d2.dindex.zip (568 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-i9a89c2f06366401291b4c2b10355c6d2.dindex.zip (568 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-b751aedacbbf94d56a23ced2ca7b24795.dblock.zip (512 bytes)",
"2021-03-27 16:17:53 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-b751aedacbbf94d56a23ced2ca7b24795.dblock.zip (512 bytes)"
],
"Warnings": [],
"Errors": [],
"BackendStatistics": {
"RemoteCalls": 7,
"BytesUploaded": 1080,
"BytesDownloaded": 1802,
"FilesUploaded": 2,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 5,
"KnownFileSize": 3183,
"LastBackupDate": "2021-03-27T16:17:04-04:00",
"BackupListCount": 1,
"TotalQuotaSpace": 999618043904,
"FreeQuotaSpace": 467196026880,
"AssignedQuotaSpace": -1,
"ReportedQuotaError": false,
"ReportedQuotaWarning": false,
"MainOperation": "Backup",
"ParsedResult": "Success",
"Version": "2.0.5.114 (2.0.5.114_canary_2021-03-10)",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2021-03-27T20:17:52.9639877Z",
"Duration": "00:00:00",
"MessagesActualLength": 0,
"WarningsActualLength": 0,
"ErrorsActualLength": 0,
"Messages": null,
"Warnings": null,
"Errors": null
}
}
I can’t run compare on this because it didn’t upload a dlist (recall the bug I mentioned if only metadata changes), so from Duplicati’s view it didn’t even create a backup. But it does show how it is possible…
Thanks for your help ts678. Good to have the explanation and see that it is not a bug.
Well, it’s not entirely known what happened in your case, and the known bug that the dlist didn’t upload when only metadata changed raised the confusion. I do still wonder why there’s av ModifiedFolders statistic, and what it’s supposed to count. But your specific issue could certainly be explained by a modified folder like I did.