Duplicati 2.0.4.5_beta much slower than 2.0.3.3_beta

There might be a performance issue with the default filters:

Can you try temporarily disabling the default filters to see if that improves anything?

Thanks for your reply!

Sure thing. I was going to before I posted but the 4 hour backup was running. Decided to just cancel it.

Ran the shortest backup just now with the default filter disabled and it’s back to 00:03:27. Looks like that’s the culprit.

1 Like

Thanks for confirming. Hopefully we’ll be able to figure out a way to address the performance issues with the regex filters.

I have just finished the second round of backups under 2.0.3.3. Here’s the rough stats:
Initial Backup of 3.7 TB: 15 days 11 hours
Scan of this dataset of 3.7 TB (no data change): 50 mins
2nd Round of additional 900 GB (on top of the 3.7 TB): 4 days 20 hours
Scan of this dataset of 4.6 TB (no data change): 52 mins

After updating to 2.0.4.5:
Scan of this dataset of 4.6 TB (no data change): 1 hour 32 mins

I do not have filters or exclusions enabled at all; this is done on a Debian server powered with a Xeon E3-1245 (same as Sandy Bridge i7 2700)

This part seems to be taking longer than it should:

  • Dec 7, 2018 3:16 PM: Starting - ExecuteScalarInt64: SELECT COUNT(*) FROM “Block” WHERE “Size” > 102400

  • Dec 7, 2018 3:16 PM: ExecuteReader: SELECT “Key”, “Value” FROM “Configuration” took 0:00:00:00.000

  • Dec 7, 2018 3:16 PM: Starting - ExecuteReader: SELECT “Key”, “Value” FROM “Configuration”

  • Dec 7, 2018 3:16 PM: ExecuteReader: SELECT “Key”, “Value” FROM “Configuration” took 0:00:00:00.000

  • Dec 7, 2018 3:16 PM: Starting - ExecuteReader: SELECT “Key”, “Value” FROM “Configuration”

  • Dec 7, 2018 3:16 PM: ExecuteScalarInt64: INSERT INTO “Operation” (“Description”, “Timestamp”) VALUES (“Backup”, 1544166987); SELECT last_insert_rowid(); took 0:00:00:00.064

  • Dec 7, 2018 3:16 PM: Starting - ExecuteScalarInt64: INSERT INTO “Operation” (“Description”, “Timestamp”) VALUES (“Backup”, 1544166987); SELECT last_insert_rowid();

  • Dec 7, 2018 3:16 PM: Starting - Running Backup

  • Dec 7, 2018 3:16 PM: The operation Backup has started

Addendum:
A reboot of the server made it worse; the status bar and logs are stuck at the “Starting Backup” phase and there is no way to stop the backup.

I am rebooting the server again and then recreating the database from scratch to see if it helps. I have backed up the 2.0.3.3 database and configs already so if this still does not work out I’ll downgrade to 2.0.3.3

Is it correct that it is not possible to easily downgrade from 2.0.4.5_beta to 2.0.3.3_beta in order to avoid the super slow backups until this can be fixed? (I did not create a backup of the database before upgrading)

I’m running into this issue as well, I actually thought my install was somehow broken because I’m having processes that used to take 1-2 minutes (its only a 60gb backup set on a fairy fast SSD with 24 cpus available and mostly idle) take over 24hr with no foreseeable progress being made.

Things such as verifying files that used to take a few seconds to run now take 45min-1hr+ just to start…?

I’m not sure what everyone is talking about with the default filters, on my backup sets I have every filter disabled except for 3 file/folder name exclusions and I don’t see any other default options set, etc… if someone could explain what they mean by default filters, I’d gladly try and disable them (if enabled) to see if the issue goes away.

For now I kinda am just stuck in the water with 3 systems that now can’t seem to backup due to how long it takes to do so and I’m mystified at why after the upgrade…

For the record this is under Windows 10 right now.

Unfortunately yes, that is correct.

See if you’ve got anything like in this image:

Note that if you do and you disable it then your backup is likely to start including a bunch of guess it used to ignore so might still be slow…

It looks like the existence of a regex filter can severely impact performance. Of the built in excludes, the “Cache Files” group (which is included in the “Default excludes” group) has a regex filter. One thing you can try is to disable the “Default excludes” group, and include all other groups except the “Cache Files” group. This will hopefully improve performance at the expense of additional files being added to the backup.

I can confirm that I don’t have any default excludes enabled based on that screenshot and I have no regex either, only 3 “exclude folders whose names contains”, but its also only around 160,000 files in total as well.

I’m tempted to try the 2.0.4.6 build to see if the issue is resolved there…?

A major bummer

What version were you using previously?

Would have been following the default auto update channel, so 2.0.3.3 previously.

If you can survive with the slower backup a little while longer, consider holding off on trying 2.0.4.6 a bit as some people have reported issues with installing it (I haven’t confirmed those) and the UI in IE (confirmed and fixes already in testing) after updating.

I am having this problem also, but the effect is not the same for all backups.

I have 3 remote locations all backing up back to the home office. only one location saw a massive increase in time for backups. it increased from ~4 hours to ~11 hours. The other sites went from ~2 hours to ~5 hours. The size of the data is close to the same at all the sites, but the one with the massive increase in time has almost 3x the number of files. So, even before the 2.0.4.5 release, filtering more files took more time. It appears that the regex problem magnifies this greatly.

That would make sense as the regexes would have to be checked more times.

Thanks for sharing your comparison numbers!

I also upgraded from 2.0.3.3 to 2.0.4.5 on a linux server. Performance drop is quite large. Before upgrade 1.30h after 2.30h.

I wonder if there are several potential things going on:

  1. The slow regex filters discussed here.
  2. When concurrent processing was introduced in version 2.0.3.6, several users had performance issues related to file verification, which may be related to the size of the database. More details here. Some of these issues were later addressed, but apparently the issue still exists for some users.
1 Like

Looks like this is still not fixed in 2.0.4.23 beta! At least here for Mac user, unacceptable performance hit when exclude filter group items are selected.

2.0.4.23 has only one change from 2.0.4.5 - a warning for Amazon Cloud Drive users. It’s unfortunate that it was given this version label.

Looks like this may have been resolved in 2.0.4.10, according to Filter groups code is extremely slow · Issue #3395 · duplicati/duplicati · GitHub

If you are willing to go to a Canary release, version 2.0.4.22 contains this and other fixes not included in 2.0.4.5 (or 2.0.4.23).

Although I’m not sure how good overall 2.0.4.10 is (I’d favor 2.0.4.22 - see below), speed boost was:

Release: 2.0.4.10 (canary) 2018-12-29

  • Improved performance of filters by around 10x, thanks @warwickmm

Release: 2.0.4.21 (experimental) 2019-06-28 would be an alternative, if the idea of canary feels risky.

This is a cumulative release for more than 6 months of hard work.
Most of these are speedup improvements, error handling and general quality fixes.

and above hints at long intervals between beta releases. Fixes don’t help much until they’re released. Above Experimental didn’t make beta due to an FTP bug and test need. Someday new Beta will ship. Release process improvements are also being proposed to see if the release rate can be increased.

Release: 2.0.4.22 (canary) 2019-06-30 has more fixes and runs well, but stability of canary may vary.
Settings in Duplicati can change your update channel, if you don’t want to risk a worse one arriving…

Whether Experimental or Canary, it’s a move forward, due to DB upgrades. If you revert, do it quickly.

1 Like

Simply to mirror my comment on the github issue:

I had some issues downgrading from 2.0.4.23 (which I thought would have included the changes in 2.0.4.10 but does not as I learned here) to 2.0.4.21 which is experimental and includes the fix for the slow filter groups.

For information to anyone else have the problem, I had to fully uninstall Duplicati, then re-install with the installer (on Windows) before it worked. (Trying to just install over the previous install lead to Duplicati not starting at all anymore in my case)

I can confirm that the speed went back to normal while using default filters with 2.0.4.21 as compared to 2.0.4.23 (and 2.0.4.5).