Critical issues with the current canary [2.0.3.9]

Releases
1

I would like to move on a new beta release, but I think we have some lingering issues with the current canary build.

@JonMikelV: do you have an overview of what issues we need to address?

I think the performance issues were fixed, but I am not sure.

New list with the checkbox plugin:

Processing files which are not in backup sources (only if --snapshot-policy is on)

Counting of size of file to go · Issue #3305 (as Pectojin suggested)

Skip-files-larger-than ignored (no default B, KB, MB, etc. size)

UNIQUE constraint failed with SIA (Support Request) [2.0.3.6] (resolved by downgrading, I also had this with SFTP)

Backup unusable: Duplicati tells me to repair, but repairing doesn’t work (getting a few of these missing files reported, including on my SFTP - I think it’s dlist files referencing deleted or empty dblocks so repair AFTER deleting all dlists seems to resolve it)

Metadata was reported as not changed, but still requires being added? (I’ve had this randomly happen starting with 2.0.3.6 but can’t seem to force it)

There’s also misc. non-canary stuff like:

Throttle translation is inversed in French
Linux “Failed to process metadata” errors on CIFS files related to xattr change

3 Likes
2

I would add this one to the list Counting of size of file to go · Issue #3305 · duplicati/duplicati · GitHub

It appears to just be cosmetic, but it’s pretty unsettling for users.

1 Like
3

I think there are some more, but at the moment the ones I can think of are:

There’s also misc. non-canary stuff like:

2 Likes
4

I believe the issue with slow database queries might still be unresolved? It appears that some of the queries may scale poorly with the size of the local database. See my later posts for more details.

Unfortunately, I am no longer able to replicate this issue as I have since modified my retention settings and trimmed my local database size from over 4GB to a few hundred MB. This has reduced my backup time from over 20 minutes to less than 4 minutes.

5

Thanks! I will use it as my “todo” list for now.

I have two minor things, that I will fix first, and the I will go on to this list.

2 Likes
6

Hi kenkendk,
The Throttle translation has been corrected in French and several hundreds other parts have been translated, please include them in the next beta release.
Cheers,

2 Likes
7

Yep, I have pulled translations for the languages and will include them!

8

I may have found a repeatable scenario to trigger this. It seems like whenever I do multiple database repair / rebuild tests around dindex / dlist failure scenarios I eventually run into the “Metadat awas reported as not changed, but still requires being added?” issue.

And it just so happens I triggered it on a test backup yesterday so I’ll let that sit as a test bed for when new version comes out.

I love the checkbox plugin!

Though I did notice I can change the checkboxes without editing the post and it appears it’s not a wiki…hopefully it’s only because of my access and not everybody can do it.

9

True. I cannot make any change to the check boxes, they are unclickable.

2 Likes
10

It looks like there are several users that are having 404 issues with 2.0.3.9 and the Google Cloud Storage backend:

I am able to replicate this, and have a potential fix (it was a bit difficult to bisect, since many of the commits do not build). I’ll test some more and submit a pull request if things look good.

edit: pull request submitted…

11

Latest Canaries are not fun to use. Disk2disk backup, many files and >1TB with just some small MB of change, takes now hours instead of <10 minutes. Despite rebuild of database and use of check-filetime-only. Duplicati reads whole file contents (I see a permanent IO of >80MByte/s) which is terrible, because effectively I cannot run this backup any more since May.

Also, but this is probably not a problem, 10000s of warnings (# of files).

12

Hi @Tapio

–check-filetime-only for canary from the big 2.0.3.6 rewrite through 2.0.3.9 can cause warnings and slowdowns.
The “Metadata” topic above describes how the option regressed, and has speed comparisions with and without.
2.0.3.6 was Apr 23. Any May problem might have come from 2.0.3.6 picking up an existing --check-filetime-only.
Removal might not fix every performance or read rate worry. I’m just giving a specific workaround that may help.

13

Thanks. Yeah, I remove the checkfiletime flag and it still reads my whole contents. Backing up my few MBytes will maybe take 5 hrs now. This time I will try to let it finish the process, maybe on next backup it will not read everything again then.

14

Ok. There are now three open issues left on the list, and I think the snapshot issue is causing the UNIQUE constraint issue.

I think the snapshot issue is fixed, but I was never able to reproduce locally, so it may still lurk. If I am correct, this means we only have the “repair does not work” issue left.

I have some ideas for how we can make the repair process more robust, but it will take some rewriting to complete. I propose that we make a new canary with the fixes already in there, and then make the “repair does not work” issue the top priority for the next release. And we get some feedback on the snapshot issue.

1 Like
15

Sounds like a great plan (full disclosure, I was thinking of proposing the same thing). :slight_smile:

Thanks for all the effort to get these initial items resolved!

16

New version out :slight_smile:

3 Likes
17

Great fixes!
Rolling update to one server/day to confirm that all nasty bugs are gone especially this one Processing files which are not in backup sources · Issue #3307 · duplicati/duplicati · GitHub

1 Like
18

I think this is also somewhat important:

19

This thread is a bit stale - can you confirm what canary version you’re seeing this on and I’ll look at starting a new topic.

20

All of them since 2.0.3.6, will test again on latest just to be sure

Edit: I was already testing on 2.0.4.4 without having realized