High memory usage on mono-sgen

I’m using Duplicati now for a few months, and notice a gradual increase in memory usage of mono-sgen. Over a period of three weeks it is increasing gradually up to 2GB. If I restart Duplicati, the same behavior is repeated.

I’m using a daily back-up to Storj DCS with client side encryption of a share with around 400GB of data. There are no issues with these back-up runs.

Current version : 2.0.6.3_beta_2021-06-17
Mono version : 6.12.0.122

Can someone help out to find the cause of this issue?

I searched some other threads in the forum about suspected memory leak. One possible cause of high memory usage is related to your remote volume size. Did you customize it by chance (default is 50MB)? Also have you customized any options related to threads/concurrency?

I didn’t change any of these settings.

Ok, good to know. You have your destination type set to “Tardigrade” right?

Yes, using access grant. Daily AES encrypted back-up with 50MB volume size and smart retention. No special options set other than email after all operations.

Hi.

Same pb for me :> 1GB+ RAM usage
40GB datas in Stoj (Tardigrade connector) + mail notification

Stoj run in a linux docker container

Any workaround ?

Welcome to the forum @wpilon

It might be coincidence, but with two reports of this on Tardigrade, I’ll ask @TopperDEL for ideas.
The last change I know of was 2.0.6.100_canary_2021-08-11 rename of Tardigrade to Storj DCS.

Possibly this could also be debugged generically by a volunteer who can debug memory on Linux.
Volunteer developers are very scarce these days, and I wish the community could offer more help.

I suppose a useful test would be to see how, for example Backblaze B2 does on memory growth,
assuming yours is the same as OP and is growth over time. You’ll pay something, but it’ll be little.

One wrinkle might be how to find memory to test this while also keeping your existing backup run.

EDIT:

Backup to local folder might give a no-storage-fee way to see if growth also happens in that case.
Lacking the right people and tools, people seeing growth might need to try some what-if-testing…

Both backend use technically the same Implementation, so the rename should not be the reason for anything here.
With the rename came also a new Version of the underlying uplink.net and uplink-c - so this might bring different behaviour.

Did I understand correctly, that you are running it on a Server that is “always on”? Maybe there is a memory leak in my lib that would Show up in that case.

But there is also a PR open to use the latest uplink.net which should have a better memory footprint. Have to check what’s the Status with that one.

Any further invrstigation/comparison is greatly appreciate as Im lacking time for the next week, unfortunately.

Hi, good to see some activity on this topic. I still have this issue. My system is running 24/7, with daily backups. If one likes me doing some data collection or debugging, let me know. I however lack any experience with this, so detailed instructions are welcome. I have telegraph and influxdb running if this is of any use to collect metrics on this issue.

In case it matters, the original post was on 2.0.6.3, and I’d like to know what @wpilon is running.

Does uplink.net use mono-sgen? Mono-sgen is the process that is allocating increasing amounts of memory. On my system Duplicati is the only package using mono.

Ignore the forum autolink wrong guess. I think this is the C# library and it would run under mono:

uplink.NET (NuGet)

https://github.com/TopperDEL/uplink.net (GitHub site)

A .Net-wrapper to connect to the storj v3 network.

Preparation talks about the native libraries under it. I’d guess all of this is in mono-sgen memory.
I’m not sure how to tell whether memory growth is from the managed side or native side though.
Even though we got the attention of the uplink.NET developer, is info on other storage possible?
There are some other memory growth issues in GitHub Issues, but the first one I opened was on

High memory consumption on mono-sgen #4589

Storj DCS

and it says there are some forum reports too (but doesn’t link, so a search would be necessary).
Mentioning that issue should alert its participants, so maybe they can help find the similar topics.