S3 backup (Minio) - remote file list misses first character

roelkw · August 30, 2021, 1:27pm

I’ve been a happy user of Duplicati as a docker container on Unraid, but recently I ran into a problem for which I’m looking for some assistance.

I’m backing up data from an Unraid server running Duplicati as a docker container to another local server running Minio. The backup seems to run fine, and a new bucket was created and many new files were created. However, Duplicati reports that the backup job failed, because it cannot find the remote files, apparently because it doesn’t read the first character of the file name.

Found 21 files that are missing from the remote storage, and no files with the backup prefix duplicati, but found the following backup prefixes: uplicati

When I check the Minio server, all files are present and have the correct name, starting with duplicati (and not “uplicati”, without a “d”).

However, the remote log shows the following:

[
{"Name":"uplicati-20210830T131634Z.dlist.zip.aes","LastAccess":"2021-08-30T14:17:38.584+01:00","LastModification":"2021-08-30T14:17:38.584+01:00","Size":191373,"IsFolder":false},
{"Name":"uplicati-b1125a4cec853487a83139de60237e576.dblock.zip.aes","LastAccess":"2021-08-30T14:17:38.312+01:00","LastModification":"2021-08-30T14:17:38.312+01:00","Size":52391213,"IsFolder":false},
{"Name":"uplicati-b40cfdda918a84def9f1ce11101c3338e.dblock.zip.aes","LastAccess":"2021-08-30T14:16:56.728+01:00","LastModification":"2021-08-30T14:16:56.728+01:00","Size":52396845,"IsFolder":false},

The remote log also shows the following for the put commands:

* Aug 30, 2021 3:17 PM: put duplicati-i50036ed7e81e49b2a16877e7d086f844.dindex.zip.aes
{"Size":111725,"Hash":"l7L1jED6kerbpOWWrBCQokTCfdAqedISzN1UMJbw3uc="}

So that doesn’t seem to be the problem.

Deleting the job and creating it again doesn’t solve the issue.
My Minio server is running on Ubuntu 20.04, but I got the same issue with Minio running on FreeNAS (built-in service).

Any thoughts? Thanks!

drwtsn32 · September 1, 2021, 4:29am

Hello and welcome! What a weird issue. If you don’t get any other responses, I can try setting up a test environment to see if I can reproduce the issue.

Dirvann · November 22, 2021, 2:36am

I have exactly the same issue. Ubuntu 18.04 Minio backend. It cuts the first letter of the name for some reason.

Nov 22, 2021 3:33 AM: list
[
{"Name":"uplicati-20211122T023302Z.dlist.zip","LastAccess":"2021-11-22T03:33:09.561+01:00","LastModification":"2021-11-22T03:33:09.561+01:00","Size":1801,"IsFolder":false},
{"Name":"uplicati-bb80d5c220e38416c9e8c3375372f344d.dblock.zip","LastAccess":"2021-11-22T03:33:09.452+01:00","LastModification":"2021-11-22T03:33:09.452+01:00","Size":11228125,"IsFolder":false},
{"Name":"uplicati-i68c29439260f497bb13e71e3f1dc0ebe.dindex.zip","LastAccess":"2021-11-22T03:33:09.539+01:00","LastModification":"2021-11-22T03:33:09.539+01:00","Size":10656,"IsFolder":false}
]

Nov 22, 2021 3:33 AM: put duplicati-20211122T023302Z.dlist.zip
{"Size":1801,"Hash":"w5hc4kh3hzqNPZRxZYR0Sgh7AjvqOgDxCkbp2CC0tAI="}

Nov 22, 2021 3:33 AM: put duplicati-i68c29439260f497bb13e71e3f1dc0ebe.dindex.zip
{"Size":10656,"Hash":"cuBC/WVTb3BQHplDfkSj1d4rZoK50jeywGeIUzHYK+8="}

Nov 22, 2021 3:33 AM: put duplicati-bb80d5c220e38416c9e8c3375372f344d.dblock.zip
{"Size":11228125,"Hash":"cIgK2IwxjPWZ+1ezHAFjKgOuIJ8vJzOr9CkBALYxl6I="}

Dirvann · November 22, 2021, 4:15am

I do also have to mention this error only appears when a subfolder is selected in the minio bucket. When the root folder of the bucket is chosen, the issue does not appear

drwtsn32 · November 22, 2021, 5:10pm

Are you possibly using backslashes in your path in the Duplicati config? For S3 and most other back ends you want to use forward slashes.

ts678 · November 22, 2021, 9:13pm

sometimes no slashes are necessary, for example on a single-level subfolder you can just type folder.

If you want to see what an S3 URL should look like:

S3 Compatible

s3://bucketname/prefix

and you can compare that to Target URL on the Commandline screen. I “think” the only double slash is supposed to be the one at s3://, anything in the middle is single forward slash, and I’m not sure of end.

You can also take that URL (Export As Command-line can also be used if you want it quoted for you) to:

Duplicati.CommandLine.BackendTool.exe to see if you do operations without losing the first character…

fixitmjc · September 1, 2022, 9:32pm

I’ve just had the same issue where for every Duplicati file, the first letter of the filename - “D” was showing as missing. The actual files are fine and in the expected place.
I’m using an Raspberry Pi arm64 OS with IDrive-e2 S3 storage, and the cause does seem to be using a Folder path below the initial folder level.
By mistake I set the folder path to be two levels down and the files were not locatable due to the missing first letter.
I used an S3 client to move the several thousand files (!) up to a first level folder and the client then found and verified them.
This would seem to be a bug and I will try to generate a report in due course.

gpatel-fr · September 2, 2022, 6:03am

while I have not done a serious study of this problem, it would seem that this code could be a candidate:

github.com

duplicati/duplicati/blob/666b2281032460254839fdc3b6e1055fdf7ce1db/Duplicati/Library/Backend/S3/S3Backend.cs#L343


      
          

          

          public IEnumerable<IFileEntry> List()

          {

              foreach (IFileEntry file in Connection.ListBucket(m_bucket, m_prefix))

              {

                  ((FileEntry)file).Name = file.Name.Substring(m_prefix.Length);

          

                  //Fix for a bug in Duplicati 1.0 beta 3 and earlier, where filenames are incorrectly prefixed with a slash

                  if (file.Name.StartsWith("/", StringComparison.Ordinal) && !m_prefix.StartsWith("/", StringComparison.Ordinal))

                      ((FileEntry)file).Name = file.Name.Substring(1);

          

                  yield return file;

              }

          }

          

          public async Task PutAsync(string remotename, string localname, CancellationToken cancelToken)

          {

              using (FileStream fs = File.Open(localname, FileMode.Open, FileAccess.Read, FileShare.Read))

                  await PutAsync(remotename, fs, cancelToken);

          }

there is here a rather weird stripping of the first character of the file name, and it’s apparently a workaround for another bug - in a general way, this is very often a cause for other bugs I would say.
If you can compile Duplicati, maybe you could try to remove 339-343 just to check if the bug in Duplicati 1.0 (!!) is no longer relevant.

WolfTongue · October 6, 2022, 5:08pm

I got this message too using the docker version. After seeing this post I checked out the paths and could find my mistake (maybe it helps someone):

In the web ui I entered as folder “/backupname”. This worked for the initial backup, but failed at the next run with the very same error message. So I switched it to “backupname” and now it seems to run.

So my advice is:

do not enter a leading slash
also please make it water proof in the code

Larz · November 13, 2022, 12:47pm

Agreed. This is a bit of short sightedness in the code, aka a bug. (Bug is not a dirty word)
I have to remove the leading slash. This is acceptable as a short-term work-around, but the parsing needs to be reworked in the code.

ts678 · November 14, 2022, 2:09pm

No files with the backup prefix duplicati, but found the following backup prefixes: uplicati #4711
might be the open issue on this, awaiting a developer and a release. Both are scarce recently.
The forum is not a bug tracker. If anyone can look further into this, and do a PR, please assist.
Duplicati only exists and improves thanks to volunteers doing code, testing, docs, and support.