Simply to mirror my comment on the github issue:
I had some issues downgrading from 2.0.4.23 (which I thought would have included the changes in 2.0.4.10 but does not as I learned here) to 2.0.4.21 which is experimental and includes the fix for the slow filter groups.
For information to anyone else have the problem, I had to fully uninstall Duplicati, then re-install with the installer (on Windows) before it worked. (Trying to just install over the previous install lead to Duplicati not starting at all anymore in my case)
I can confirm that the speed went back to normal while using default filters with 2.0.4.21 as compared to 2.0.4.23 (and 2.0.4.5).