Hi. I’m a new user (Crashplan refugee), just a couple weeks now. Platform: laptop, Fedora, XFCE. As a general proposition, I’m loving Duplicati a lot (using some spare room on my own website for storage via FTP, hence “free” because website already paid for).
But I’ve found a couple of hacks that help me a lot. If Duplicati supported these in better/clearer ways (maybe as advanced settings, or even just more documentation?), it might help others too.
#1. When Duplicati runs, nominally in the “background,” it was using too much CPU and disk IO, hence affecting my foreground work (you know, jerky mouse and keyboard, window painting, stuff like that). So I now launch(/autostart) it like this: “nice -19 ionice -c idle duplicati”. That makes a world difference. It now runs very quietly out of the way, just like it should.
#2. Sometimes, both when backing-up and when restoring, Duplicati was causing excessive memory usage, and even forcing some swapping! Shouldn’t do that (as I had 4GB free mem before Duplicati runs). Thanks to another thread in the Duplicati Forum, I’ve not moved my TMPDIR environ var, from /tmp to $HOME/.TMPDIR. The jury’s still out on whether this fix will be as dramatic as #1, but so far it’s looking good.
Neither of these performance hints touches on end-to-end execution time of backups or restores. I know some folks care about that, but I don’t, because I’m not using Duplicati as an IT for enterprise purposes, just an individual user, hence that kind of perf isn’t a bid deal for me. Except for the two perf-killing hints mentioned, all I really want out of Duplicati are functionality/reliability. And I think I’m going to be happy about that.
Correct, /tmp is tmpfs, 4G. Never been an issue until Duplicati (well, together with whoever else wants to use it at the time, but they all play along nicely with one another).
Need to ask: There’s a “–tempdir” advanced setting (command-line option) for Duplicati, but that doesn’t work to fix my problem. Is it broken, or does it mean something else?
When you get low on RAM again, check to see if there are any big files in the /tmp or /var/tmp directories. If there are any duplicati based files, tell us which file it is so we can determine where you issue may be. Also check to see if the large memory usage is the duplicati process itself.
I have a feeling that there may be some configuration setting issue, just trying to narrow which one.
Oh, OK, cool. I try to find some time to revert my TMPDIR and see if I can reproduce the behavior. Any special settings for logfiles you’d like me to set?
Hydrian, before going more deeply into debugging, let me show you a typical failure I’m seeing (this one on restore, but I’m also seeing them on backup). Maybe this’ll give you some ideas of how to proceed.
See the attached ZIP containing screenshots. (Your forum won’t permit TGZ.Screenshots.zip (536.2 KB)
)
#1 shows steady-state. Life is good. Note the CPU and memory readings, shown in both top (sorted by MEM) and gkrellm (along RHS). Duplicati shows up in the background/weeds (at nice/low priority 39), like it should. (Ionice is also set at idle, but that’s not visible, or interesting for us.) Note the timestamps at the top of gkrellm, for future reference.
#2 shows me requesting a restore.
#3 shows Duplicati starting to think about it. Note the Duplicati CPU showing up in gkrellm, in light green (which means “nice”). So far, so good.
#4 shows a few seconds later. More Duplicati CPU, but it’s nice, so no problem.
#5 shows a few seconds later, and now things are getting bad! Note the memory usage (gkrellm, bottom right) is creeping up steeply! The 3 krells in gkrellm memory show 4 sections: used|buffers|cache|free. The cache usage is going wild!
#6, a few seconds later, shows the memory continuing to rise. Memory just about gone (but no swapping).
#7 shows the failure. Note the cache memory has been freed.
I suspect the “Insertion failed because the database is full database or disk is full” message (hmm…might need to re-word that) is coming from SQLite as mentioned here in which case you might need to set the TMPDIR environment variable:
It’s also possible the database file in question is being stored on a full partition, though normally I’ve only see that with Dockers and Synology NAS devices:
As for /tmp usage, keep in mind that Duplicati will create up to 4 dblock (archive Volumes) in preparation for upload, so if your “Upload volume size” is large (say 500MB), you could be running out of temp space due to needing 2GB for waiting-to-be-uploaded files.
If that’s the case, you can resolve it by setting a smaller --dblock (“Upload volume”) size and / or adjusting the --asynchronous-upload-limit:
--asynchronous-upload-limit (default 4)
When performing asynchronous uploads, Duplicati will create volumes that can be uploaded. To prevent Duplicati from generating too many volumes, this option limits the number of pending uploads. Set to zero to disable the limit
I was sort of going down the same path as @JonMikelV was. Though you at the start of process you had over 5 GB free of space (if you sacrified buffers and cache).
@wtuvell Did you change the ‘db size setting’ or the number of uploads?
As noted previously, I’d been setting TMPDIR = $HOME/.TMPDIR (>100G), and it seemed to help, but I reverted to /tmp (4G, mostly all free) for this test. I did previously (few days ago) monitor, but nothing seemed suspicious there, though I wasn’t looking all that diligently, I’ll try looking again. Haven’t checked /var/anything.
My database is stored in $HOME/.config/Duplicati, total of 2.8G used in that dir, >400G free according to df command (/dev/mapper/fedora-home).
I have not (yet) touched the settings you guys mention, will think about experimenting with those too (one var at a time thought, scientific method). Right now I’m just using:
Hmm, something fishy. I tried the same restore operation as before, got the same failure (basically, OOM). But this time I monitored $TMPDIR = /tmp (console printout below;). It shows no irregularities, in fact, it seems to show /tmp not being used at all! That is, the conjecture about “running out of temp space (on disk)” doesn’t seem to be happening.
Hmm, an off-the-wall thought. In that sample failure run I posted (ZIP file), it showed memory usage increasing until the 8G RAM limit was hit, and then the Duplicati operation (list in preparation for restore) failed.
The problem I don’t understand is this: All the mem/RAM was not really “used,” it was just “in use.” That is, all the 8G wasn’t really being used/required by applications. Rather, there was zero “free/usable” mem remaining, but there was a ton of mem still sitting, “unused,” in cache.
Why didn’t the kernel free that cache, and move it to the free arena, so that processes (Duplicati in this case) could use it, instead of killing Duplicati (presumably for OOM).
After all the cache is there just for performance enhancement. It’s not really needed in RAM. It could be moved to swap, or even deleted altogether and regenerated when needed later.
So, is this a kernel bug? Or am I missing something? (Yeah, I know, probably the latter …)
Hmm, good guess, but no. You can see my proc usage from the gkrellm screenshots (in previous upload), and ulimit -a says:
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31375
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31375
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
(On a related note: since Duplicati has been behaving nicely for me since I made my earlier changes, as reported in this thread, I haven’t been revisiting it lately. In particular, I haven’t yet upgraded to 2.0.3.3, which I’ll do very soon.)