Backup files missing on Wasabi

Support
1

I was able to recreate a scenario where backup was successful but files are not visible on Wasabi.

I have a Linux server that was backing up personal files to Wasabi. Approximately 12 versions using “Smart”

I re-installed the server and re-installed Duplicati latest version. In setting up duplicati, I entered path as:

Bucket: personal
Filepath: /Home/Server

I expected error saying that remote files were found but did not.

Backup ran nightly for 3 days and uploaded the files, no errors.

Checking Wasabi, I could not find the new files. Existing file dates were still old.

It appears that the trailing “/” in the Folder name is a problem. When I removed the training “/” it found the existing files.

I still cannot find the files in wasabi file manager.

1 Like
2

Welcome to the forum @Stephen_Ferranti

If you have firm signs that files are going to Wasabi (e.g. their space or transfer counter is going up), probably the best people to ask for help in finding your files are at Wasabi. Or maybe they can say what’s not visible…

3

The Issue is definitely the LEADING “/” in the file path. I ran the backups for 3 days and Duplicati reported that the backups were successful.

In Wasabi, the folder is empty. I used Cyberduck to browse my bucket and there are no files. When I removed the LEADING “/” in the filepath and re-imported the backup. It worked and the files are in wasabi.

Duplicati is wrongly reporting success when no files are written to a wasabi folder that has a LEADING “/”.

1 Like
4

TL;DR jump to the bottom, if you like. There might be a bug in Amazon’s lib, but test can help confirm.

What was the answer to this, please. I don’t have Wasabi, but their knowledge base provided this page:

How do I check my Wasabi usage? and if that is not detailed enough to answer the question, they have:

How do I interpret the bucket size .csv file and the bucket size info on the Wasabi UI?

Depending on Internet speed, you might also be able to tell whether your upload was using the network.

Made me think that it was probably uploading. You can possibly run netstat to get idea of where it;s going. What I’d prefer is a tool that shows where heavy traffic is going, but I don’t know if Linux has an easy one.

If you have the backup that didn’t backup (or you can make a small one for a test) you can try some tests.

Viewing the log files of a backup job can provide lots of useful information in two primary areas of log files.

<job> → Show log (pick a misbehaving backup) → Complete log shows a long BackendStatistics report. BytesUploaded, KnownFileCount, and KnownFileSize might be some of the more relevant statistics on what Duplicati thinks it did. The question is where did the data actually go. Remote directory listing runs by default after a backup, and should check that everything looks fine. Using <job> → Show log → Remote and clicking on the list at end of backup will show exactly what Duplicati saw when it did directory listing, however it uses the sample file name because Duplicati backups are a flat (and very portable) set of files.

If you want a detailed look at how fast the files are going (wherever they’re going), the log at Profiling level (which is very wordy) shows all the timings on a per-file-upload basis Here’s a recent default-size dblock:

2020-07-29 08:46:20 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 44.69 MB in 00:01:07.6082929, 676.84 KB/s
2020-07-29 08:46:20 -04 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Put - Completed: duplicati-b106d1b0e8d844cad9ca9af6b0f14d455.dblock.zip.aes (44.69 MB)

and the speed is about what I expect for speed that I think my Internet connection has as usual maximum.

Duplicati.CommandLine.BackendTool.exe will let you do manual experiments such as directory listings of storage URL of your choice (an easy aid if it’s hard to guess format is to do an Export As Command-line).

I should also mention that Duplicati (for Beta, as of 2.0.5.1) offers these two different S3 backend libraries:

image

By default, your S3 uploads will use the AWS SDK for .NET by Amazon, whose source site is found here.
As with many GitHub projects, they have an Issues tracker. I figured I’d give it a quick search, then I found:

AmazonS3Client.CopyObjectAsync fails silently when destination key has leading slash #1388
Duplicati’s interface code actually does PutObjectAsync but maybe it shares the CopyObjectAsync bug.
On the surface, this is an interesting finding, but doesn’t explain how Duplicati verification saw remote file.

5

I will create a test tonight and upload a few files. and then try a restore. There are NO files on the wasabi drive. I looked everywhere and with 3rd party tool (Cyberduck). I can recreate the problem by adding the “/” to the folder name.

6

Please make sure the Wasabi usage stats concur, and see if you can see Duplicati list seeing files.
You can also try your own list, see upload times or download times, watch Internet activity, see logs.

New information would be helpful to understand where the data is going (and coming from, if a restore works). Restore test should set no-local-blocks=true which will force Duplicati to go to remote instead. Question then becomes where data is that Duplicati thinks is remote. It’s like upload study in reverse…

EDIT:

If somehow you can’t find a list easily, the job’s Verify files button will put one in the job’s Show log → Remote display, and will by default also download 1 set of 3 files from wherever the backup is residing.

7

On Linux, tcpdump can do this. Or wireshark (-cli).
Example for s3.eu-central-1.wasabisys.com would be:

tcpdump -i eno1 -vvv -nnn host s3.eu-central-1.wasabisys.com

Change the eno1 (see ip link command) and host to whatever suits your sitation. To log to a capture file, add -w duplicati.pcap to the commandline. You can open and read the file in Wireshark.

I’ll be watching this thread since I am on Wasabi as well.

8

Thanks for jumping in. We’ll see where it goes. Feel free to poke at it yourself if things slow down.
It would be nice to figure out where the problem is, and I can’t do the testing, having no S3 at all…

9

Ran a backup using a leading “/” in Folder Name. It was successful, I ran a
restore, it was also successful including using “no-local-blocks”. Here is
backup and restore logs.

{
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“ModifiedFiles”: 0,
“ExaminedFiles”: 3,
“OpenedFiles”: 1,
“AddedFiles”: 1,
“SizeOfModifiedFiles”: 0,
“SizeOfAddedFiles”: 7,
“SizeOfExaminedFiles”: 5543943,
“SizeOfOpenedFiles”: 7,
“NotProcessedFiles”: 0,
“AddedFolders”: 0,
“TooLargeFiles”: 0,
“FilesWithError”: 0,
“ModifiedFolders”: 0,
“ModifiedSymlinks”: 0,
“AddedSymlinks”: 0,
“DeletedSymlinks”: 0,
“PartialBackup”: false,
“Dryrun”: false,
“MainOperation”: “Backup”,
“CompactResults”: {
“DeletedFileCount”: 0,
“DownloadedFileCount”: 0,
“UploadedFileCount”: 0,
“DeletedFileSize”: 0,
“DownloadedFileSize”: 0,
“UploadedFileSize”: 0,
“Dryrun”: false,
“VacuumResults”: null,
“MainOperation”: “Compact”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:12:51.1270757Z”,
“BeginTime”: “2020-07-31T15:12:51.1270757Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 3559,
“BytesDownloaded”: 3559,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“VacuumResults”: null,
“DeleteResults”: {
“DeletedSetsActualLength”: 0,
“DeletedSets”: ,
“Dryrun”: false,
“MainOperation”: “Delete”,
“CompactResults”: {
“DeletedFileCount”: 0,
“DownloadedFileCount”: 0,
“UploadedFileCount”: 0,
“DeletedFileSize”: 0,
“DownloadedFileSize”: 0,
“UploadedFileSize”: 0,
“Dryrun”: false,
“VacuumResults”: null,
“MainOperation”: “Compact”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:12:51.1270757Z”,
“BeginTime”: “2020-07-31T15:12:51.1270757Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 3559,
“BytesDownloaded”: 3559,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:12:51.1270757Z”,
“BeginTime”: “2020-07-31T15:12:51.1270757Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 3559,
“BytesDownloaded”: 3559,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“RepairResults”: null,
“TestResults”: {
“MainOperation”: “Test”,
“VerificationsActualLength”: 3,
“Verifications”: [
{
“Key”: “duplicati-20200731T151250Z.dlist.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-i3672d56ccf844751bd9787ae089c23fe.dindex.zip.aes”,
“Value”:
},
{
“Key”: “duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes”,
“Value”:
}
],
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:12:51.5020803Z”,
“BeginTime”: “2020-07-31T15:12:51.236452Z”,
“Duration”: “00:00:00.2656283”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 3559,
“BytesDownloaded”: 3559,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
},
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:12:51.5177056Z”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:01.5074047”,
“MessagesActualLength”: 19,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.Controller-StartingOperation]: The
operation Backup has started”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Started: ()”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Completed: (3 bytes)”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Started:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Started: duplicati-20200731T151250Z.dlist.zip.aes (1.29 KB)”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Completed:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Completed: duplicati-20200731T151250Z.dlist.zip.aes (1.29 KB)”,
“2020-07-31 11:12:50 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Started:
duplicati-i3672d56ccf844751bd9787ae089c23fe.dindex.zip.aes (973 bytes)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Put - Completed:
duplicati-i3672d56ccf844751bd9787ae089c23fe.dindex.zip.aes (973 bytes)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.Operation.DeleteHandler-DeleteResults]:
No remote filesets were deleted”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.Database.LocalDeleteDatabase-CompactReason]:
Compacting not required”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Started: ()”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Completed: (6 bytes)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Started: duplicati-20200731T151250Z.dlist.zip.aes (1.29 KB)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Completed: duplicati-20200731T151250Z.dlist.zip.aes (1.29 KB)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Started:
duplicati-i3672d56ccf844751bd9787ae089c23fe.dindex.zip.aes (973 bytes)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Completed:
duplicati-i3672d56ccf844751bd9787ae089c23fe.dindex.zip.aes (973 bytes)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Started:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”,
“2020-07-31 11:12:51 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Completed:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 8,
“BytesUploaded”: 3559,
“BytesDownloaded”: 3559,
“FilesUploaded”: 3,
“FilesDownloaded”: 3,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Backup”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:12:50.0103009Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

{
“RestoredFiles”: 1,
“SizeOfRestoredFiles”: 7,
“RestoredFolders”: 0,
“RestoredSymlinks”: 0,
“PatchedFiles”: 0,
“DeletedFiles”: 0,
“DeletedFolders”: 0,
“DeletedSymlinks”: 0,
“MainOperation”: “Restore”,
“RecreateDatabaseResults”: null,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “2020-07-31T15:13:24.7592238Z”,
“BeginTime”: “2020-07-31T15:13:24.2582828Z”,
“Duration”: “00:00:00.5009410”,
“MessagesActualLength”: 7,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: [
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.Controller-StartingOperation]: The
operation Restore has started”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Started: ()”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: List - Completed: (6 bytes)”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.Database.LocalRestoreDatabase-SearchingBackup]:
Searching backup 0 (7/31/2020 3:12:50 PM) …”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.Operation.RestoreHandler-RemoteFileCount]:
1 remote files are required to restore”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Started:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”,
“2020-07-31 11:13:24 -04 -
[Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend
event: Get - Completed:
duplicati-b08d50c79723a41c69608663088d52f3b.dblock.zip.aes (1.23 KB)”
],
“Warnings”: ,
“Errors”: ,
“BackendStatistics”: {
“RemoteCalls”: 2,
“BytesUploaded”: 0,
“BytesDownloaded”: 1261,
“FilesUploaded”: 0,
“FilesDownloaded”: 1,
“FilesDeleted”: 0,
“FoldersCreated”: 0,
“RetryAttempts”: 0,
“UnknownFileSize”: 0,
“UnknownFileCount”: 0,
“KnownFileCount”: 6,
“KnownFileSize”: 2987806,
“LastBackupDate”: “2020-07-31T11:12:50-04:00”,
“BackupListCount”: 2,
“TotalQuotaSpace”: 0,
“FreeQuotaSpace”: 0,
“AssignedQuotaSpace”: -1,
“ReportedQuotaError”: false,
“ReportedQuotaWarning”: false,
“MainOperation”: “Restore”,
“ParsedResult”: “Success”,
“Version”: “2.0.5.106 (2.0.5.106_canary_2020-05-11)”,
“EndTime”: “0001-01-01T00:00:00”,
“BeginTime”: “2020-07-31T15:13:24.2582828Z”,
“Duration”: “00:00:00”,
“MessagesActualLength”: 0,
“WarningsActualLength”: 0,
“ErrorsActualLength”: 0,
“Messages”: null,
“Warnings”: null,
“Errors”: null
}
}

10

Although perhaps there’s a typo somewhere, and the latest test didn’t note if Wasabi files are “missing”, there’s at least some surface inconsistency possible. I don’t have all details, but does it seem that way?

I’m glad the latest backup and restore worked OK. I don’t know if the original issue restored OK, despite invisibility of the backup files that may or may not have gone to Wasabi. There were tests to look at that.

Clearly from restore testing now, the data is somewhere, but exactly where would need the other exam.

The log looks entirely reasonable for a small backup. You upload 1 dblock, 1 dindex for it, and job’s dlist.
How the backup process works details more, but dblock is file data, dindex indexes it, dlist lists all files.

Verifying backend files downloads one set of 1 dblock, 1 dindex, and 1 dlist. Only oddity is that it reports:

“KnownFileCount”: 6

even though it did

“FilesUploaded”: 3

however

“BackupListCount”: 2

makes me think two backups were done, and the math might have been that each one uploaded 3 files.

What next? I’m still hoping for requested test data about the invisible file issue, and there’s another user.

11

I updated the support case with wasabi. I can’t find the files anywhere. I used cyber duck to browse and still can’t find them.

12

Cyberduck decided they’re OK with having some invisible files, and Wasabi manager might be similar:

#6757 closed defect (wontfix) Keys with double slashes are invisible

I think this stems from the fact that there are not truly folders, but a folder simulation based on slashes.
The double slash case would (I think) not be kept on your local filesystem, e.g. Linux treats it as single:

$ mkdir -p /tmp/foo//bar
$ ls foo
bar
$

Object key and metadata places very few rules on what’s in the key beyond that size limit is 1024 bytes, however it does discuss the use of a delimiter (commonly “/”) to create the illusion that there are folders.

Folder-oriented viewers could typically break paths apart at path delimiter to see pseudo folder structure. Question for the double-slash case becomes how to display the nameless folder level implied by double slash with no “folder” name between. It could be shown as a nameless level, but easy path is to ignore it.

Their convention seems to be to not use what an OS would call an absolute path beginning with a slash, therefore a leading slash might get how’s-the-folder-before-that-slash-shown? Trailing might also double because Duplicati will probably add a slash before adding what would be a filename on any ordinary OS.

I suggested numerous ways to infer whether or not the files are on Wasabi, but for direct view you might need some lower-level viewer, preferably one that will let you see all of the keys in the bucket without any attempt to “folderize” the view. If the tool doesn’t do it, it can’t do it wrong. I mentioned Duplicati’s tool, but
Quick way to list all files in Amazon S3 bucket? seems one way to do it, and there are many other ideas.

AWS CLI User Guide.if you want to use Amazon’s tool to look around.

13

Hi,

I used the Minio (https://github.com/minio/mc) client and was able to find the files, they were in fact in a directory named “//” that was not visible in Cyberduck or WebGui. I was able to delete all the files in that folder using the mc tool.

Thank you for all your help.
-Stephen

3 Likes
14

Thanks guyz It’s very useful !

15

@Stephen_Ferranti, thanks for the helpful and detailed posts on the issue and for posting the solution. I do have a few simple questions which might be obvious to the observant reader, but it might also confuse others. It might be a good idea to go through your posts and correct them.

There is no setting called “Filepath” that I can find, at least not in the GUI. I asume you mean “Folder Path” on the page called “Backup Destination” when setting up or editing a backup?

@ts678 hinted at this as well, in one post you talk about a trailing “/”, and elsewhere its a leading “/”.

For completeness and clarification, can you please state which one led to your issues?

In other posts where you speak of the full path they start with a double “/” and it suggests its the leading forward slash that led to this issue but it might still confuse people.

So, with the leading "/’ in “Folder Path” backups and restores are succesful, its just that file browsers like Wasabi’s web interface or Cyberduck are unable to see the files? Is that correct?

Were you able to test this with the minio engine from Duplicati to see how that behaves?

Thanks again Stephen!

@ts678 I see that I can edit and create wiki pages on github, is it an idea to create a “known issues” wiki page with a summary of issues and solutions, and a link to the forum thread disussing the issue?

16

It’s an idea, but seems to me like a very high maintenance way of replicating forum and GitHub search, which seem to be little used (but GitHub does sort of auto-search when you start drafting a new issue).

Duplicati User’s Manual does eventually collect notes on some common issues, and is searchable too.
Duplicati FAQ exists now but I think few people know of it or use it, and I doubt it’s periodically reviewed.

Personally, I often use Google search to search Duplicati materials when its more powerful queries help.
Clearly it’s high-content search, which is why I harp on search, and I suspect next idea may also need it:

GitHub has 910 open, and 2449 closed issues. Forum has 41.6 thousand posts, in 4.1 thousand topics.
If I were convinced that having the summary would either help people or attract volunteers to help, I’d be more inclined to say that maybe it’s worthwhile trying to create and maintain such ever-changing data…

There’s certainly development value in trying to determine which issues (if any) are even describable in meaningful ways, and possibly consolidating duplicates. These are better fix targets than are mysteries.

Or it might be multiple (and ever-more) threads, and some in GitHub too. One can’t control posts easily.

Are you interested in volunteering in any areas mentioned or others? There’s a significant help shortage.

17

And here I thought I could get away with that by donating :rofl:

Can you send me a PM that more clearly describes what I could help with?