Sure enough, it’s back to running slow:
This confirms that the problem is how something in the duplicati container is interacting with a FUSE, since everything else is the same between the two test cases. What’s not clear is 1) which software in the container is causing the slowdown and 2) why it’s causing the slowdown
While I appreciate the idea that perhaps a software change is involved, another variable is data:
The container upgrade and the database version upgrade (four new indexes) were at same time.
This makes it hard to say whether issue came from new software or new data. They interact too,
for example the same DB Browser for SQLite found that new index got used after DB added that.
Rather than manually drop an index or four, then vacuum to recover space, it might be simpler to downgrade the container to 2.0.6.3 and manually edit the Version table Version back to 11 as per
release notice, however if the container wants to autoupdate, don’t let it as it will error on indexes
already being there. In the future such index creates should probably get IF NOT EXISTS clause.
Alternatively you can stay on current container and see if you can vacuum and DROP your way to
speed even while using FUSE. That could be done a bit at a time or all at once then wait for a day.
While a bit slow, this sort of test seems more workable than tracing changes in the LSIO container.
At some point, using tools to discover things like system load, I/O wait, bytes read, etc. would help.
Doesn’t the fact that the issue goes away when we bypass FUSE preclude data from being a factor? Is the data (or more specifically, these new indices) interacting with the file system in some way that would cause it to behave differently on a FUSE than on a “normal” file system?
No, and it’s not just a FUSE thing because your smaller backup jobs work. It might be many variables.
The idea is more like there is a certain usage point beyond which FUSE grows slow. Maybe you hit that, e.g. by overflowing a cache (for Windows cache, I gave prior examples of the huge difference it makes).
How to clear the buffer/pagecache (disk cache) under Linux is the native way. I don’t know FUSE cache.
New indices add space, add some sort of hard-to-study access pattern, and make SQLite run differently.
I’m also asking for some metrics on the file system interaction which might answer the question you ask. Probably finding the exact access pattern will be too hard, but aggregate information should be possible.
You could sample Canary releases to pursue the theory that it’s software. 2.0.6.102_canary_2022-04-06 should have the next index plan, and that will take you back more than a year of LinuxServer.io changes. Coming from the other end is less effective. 2.0.6.100_canary_2021-08-11 had the old index design, but is less than two months from 2.0.6.3_beta_2021-06-17 which is what I proposed, if an old one got tested.
Maybe this is splitting hairs, but I would still call this a software issue. The data structure, including tables, views, and indices is just data. If the software controlling how you access the data is not configured to handle the size of the data, then the answer is to fix the software configuration.
That there may be a reasonable way to reduce the size of the data is beside the point. It very well may be worth exploring that possibility, but doing so without identifying the configuration (or bug!) that’s responsible for the slowdown will only kick the can down the road.
If I’m missing your point entirely, let me know… I’m always happy to learn, sincerely.
Or fix the data. As to fixing the software configuration, there’s no guarantee the issue is the container.
The part of the container that LSIO makes is probably going to be the hardest possible to investigate, especially unless some things (such as entire container) get swapped which is what I’m suggesting…
I guess we’ll see if anybody else can think of a way to isolate issue without doing more experiments…
that’s interesting debugging, thanks for that.
Now browsing through the linuxserver releases on Github
it seems that they switched to Jammy as base OS quite some time ago for the Canary versions, however they kept the old image for Duplicati beta.
So even if you refreshed your docker image, the OS did change with beta - from Focal to Jammy (20.04 to 22.04 for people not versed in Ubuntulogy).
Note that Ubuntu has switched to fuse3 as default for Jammy.
A possible way to compare could be to run the ‘official’ image (running on old Buster OS image), or even the base image with something like
sudo docker run -it mono:6-slim
copy the problematic database to this container (sudo docker cp), install sqlite3 in the container (apt install sqlite3) and run the query vs running it in the real container.
Is this in the picture though? User Shares in Unraid are where FUSE comes in. Docker doesn’t do that. Container runs on Unraid kernel and container libraries I think, but how would container FUSE be used (presumably by container SQLite)? FUSE is provided by Unraid. I’m still looking for how it’s configured. Finding almost nothing. Finding a lot of performance complaints though, and we have one to add there.
Probably not as you say.
So I finally read the release notes for Unraid 6.12 as I was preparing to update my server today. Turns out they DID address the FUSE issue. Or rather, they gave us a built-in way to bypass FUSE in the correct circumstances. For the curious, read the “Share Storage Conceptual Change” section of the 6.12.0 release notes.
Honestly, that’s good enough for me to consider this issue resolved.
After upgrading Unraid, I did set up my appdata share to be an “Exclusive Share,” which is the signifier in Unraid to bypass FUSE. I updated my Duplicati config to undo the previous workaround to bypass FUSE manually. I will report back if I experience slowness again, but I expect it will work well.