Problems using Temp in a ramdrive

Support
1

In short, is there any reason that Duplicati doesn’t like using a ramdrive for its temp directory?

Here’s the background.

By default, Duplicati uses the current %TEMP% folder for temporary storage which usually works out to be C:\Users<username>\AppData\Local\Temp and, these days, most are running an SSD as their OS drive. Observing that Duplicati is fairly ‘TEMP-intensive’ and in the interests of SSD preservation, I’ve toyed around with moving Duplicati’s TEMP to other locations.

One trial I ran was with a ramdrive. I used the ImDisk toolkit (ImDisk Toolkit download | SourceForge.net), which is widely regarded and still developed/supported (however, I just noticed that, as of Dec/24, ImDisk has been replaced with the AIM Toolkit project. I started testing before that so was unaware till now), with dynamic allocation and the drive mounted as an existing folder. The ramdrive was permanenly enabled, not spun up/down specifically for duplicati’s use.

I ran my ramdrive trial for around 4 months but recently declared defeat.

I initially used a 2GB ramdrive which was expanded to 3GB. The ramdrive couldn’t be used for restore operations since even 3GB still wasn’t enough but it was used for most other operations. Which is to say that, naturally, I was aware of space exhaustion issues and mitigated those by disabling ramdrive use for the specific operation.

The most common problem I encountered was corrupted backup databases. However, there was also cases of files missing from the remote storage, files that couldn’t be decrypted, backup operations stalling (ie simply stopping with no errors and no further log entries, as if there was a deadlock, though it responded to a Ctrl-C). Repeated repair, rebuild and simply delete-all-data-and-start-again operations ensued. I also do automated restore tests and I’ve noticed that non-critical errors have crept into those as well where some files can’t be restored (‘touching’ the file so it gets updated in the next backup resolves those).

There was never anything ‘specific’ in the errors that I could chase down, the problems seemingly occurring randomly. I have a weak but still very much unsubstantiated feeling that deleting anything (eg pruning, compacting, etc) was at the root of problems. I found that the stalling issue could generally be traced to a bad DB, despite there being no errors, and a rebuild usually fixed that. I did spend time trying to pick apart the stalling issue by inspecting the DB in an external application but quickly drowned in data. This is also why I haven’t posted about this till now: I had nothing to pin-point as specific behaviour.

To round out the saga, my ramdrive testing roughly coincided with the update from 2.0.7.103 to the 2.1.* .Net8 versions and, I have to admit, I aimed most of my frustration at the new platform, to the point that I was seriously considering going back to the 2.0.* branch. The only thing that stopped me was that I didn’t see any similar issues raised in this here forum.

Then, I just up and decided one day to kill the ramdrive. Lo and behold, things have improved.

Now, yes, I use the canary release branch so ‘caveat emptor’ is understood and accepted. I’ve been using the canary branch for years, largely without issue.

Yes, it could well be the ImDisk driver, though that seems rather poor form for a disk driver, all the more surprising considering its age and popularity. I’ve considered whether SQLite has a known aversion to ramdrive use but found nothing. While I don’t think there’s a “slap-you-in-the-face” bug in Duplicati, I do wonder if there’s some sort of interaction with the async/await calls, or perhaps there’s an overlooked missing await that only causes problems if a filesystem/DB call returns very fast. [I’m a .Net dev by day and, as a long-time avid user, would gladly donate spare time, if I had it.]

I don’t see how it should make a difference but in case it does, I do all my Duplicati’ing as Duplicati.CommandLine.exe scripts, rarely using the web UI. Besides finding scripting much more convenient, it also allows capturing detailed (versbose by default) logs much easier; not that this helped much in this situation. I have multiple backups run at various schedules stored in Minio NAS storage (though I’m still using the S3 provider) and, as mentioned above, I regularly run full restoration tests for all backups.

I realise there’s a lot to pick apart here and thank you for reading this far.

I’d love to hear of anyone else’s experience with using a ramdrive with duplicati.

Thanks

2

Do you have any examples of what these corruptions looked like?

That sounds unstable. In versions prior to 2.1.0.110 the code would attempt to decrypt the file before checking the hash, so it is possible that the file was damaged somehow, but sounds really unstable.

These are really hard to debug. Usually you would need a developer environment, like Visual Studio, Rider or VS Code set up, so you can see where all parts are stuck.

That sounds very disturbing. Do you have logs or error messages that could help track this down? If we cannot locate the source, at least we could put in a check in the code that identifies the situation you saw and alert the user.

That is my immeidate thought as well. Duplicati will produce and store temporary files prioir to upload. If these files are somehow damaged before upload, Duplicati will not notice and will happily record the damaged file hash as the “correct” hash.

That could explain some of it I think. SQLite relies on the operating system locks and attributes, so if these are not correctly emulated by the ramdisk, strange things might happen.

It is the same code that is invoked, so it should not matter.

I can’t fully rule it out, but it would be surprising. Most IO calls have some kind of cache structure to insulate against the slowness of disks, so even without a ramdisk, many calls should be just as fast.

If you want to test further the idea that it is something with SQLite and the ramdisk, you can try to set the environment variable SQLITE_TMPDIR to your regular disk, while keeping the tempdir set to the ramdisk. This should cause SQLite to store files on a regular disk, and any other files to be stored on ramdisk.

In case this is some weirdness with SQLite, the problems should go away but still reducing write-tear on the SSD.

3

I don’t use ImDisk, but forum search (top right of screen) finds four good reports. You’d have to read them to hope for details, e.g. if it was transient, had DB, etc.

ImDisk does sound like it can do persistent disk by saving, but what if it cannot? Would it tell you on next start that it only has an old version of the data around?

A less likely concern (because I heard it defaults to NTFS) is – what filesystem? ImDisk (per a page I saw) supports many (it’s probably just blocks). Some, e.g. FAT32 and maybe some other possibilities, don’t do locking as SQLite requires.

4

I’ve been using IMDisk to create a RAM disk for Duplicati for years… for much the same reasons, to save wear and tear on SSDs.

All I have done is specified the tempdir= option to point it at the mount point for the RAM disk. This causes Duplicati to store any temporary files it needs in this place, but as far as I know SQLite doesn’t use it unless you explicitly tell it to do so.

Moving the SQLite databases to the RAM disk always sounded too risky for me, so I have never tried it.

I have never experienced any problems with Duplicati that I could trace conclusively to the use of a RAM disk … aside from a few times where I accidentally ran the machine out of memory while a backup was running.

1 Like
5

Thanks for sharing your setup and results.

The help text (as shown in the old manual) clarifies what SQLite does for temp:

tempdir

Note that SQLite will always put temporary files in the system default temporary folder. Consider using the TMPDIR environment variable on Linux to set the temporary folder for both Duplicati and SQLite.

which implies Windows won’t do it at all (and I sort of recall a past forum topic).

6

Hi @kenkendk,

Thanks for taking the time to consume my post and respond.

Do you have any examples of what these corruptions looked like?

I do have a saved copy of a malformed database but it’s 1.55GB (7zip takes it down to 656MB) so, unless you’ve got a file drop or perhaps I can send you a OneDrive link or something?

These are really hard to debug. Usually you would need a developer environment, like Visual Studio, Rider or VS Code set up, so you can see where all parts are stuck.

Yeah, absolutely. I don’t have VS on my PC (I like to keep things separate and use VMs for development) but what I should have done was capture a process dump for inspection with windbg.

Do you have logs or error messages that could help track this down? If we cannot locate the source, at least we could …

I kept many logs that have problems but, in most cases, there’s not much detail other than the discovery of the problem. Most logs of no interest are simply overwritten by subsequent runs. I do have logs that capture the removal and apparent repeated resurrection of a extra remote file and some other entries of interest. I’ve collected the relevant information, with notes, into the attached ‘Duplicati_phoenix_file.txt’, search for ‘b1b862e7f1ad448c0afbd7e653dc71694’.
Duplicati_phoenix_file.zip (6.4 KB).

I think I might set up a small’ish sacrificial backup just to capture details of these problems.

7

I did read those but they’re mostly c.2018 so didn’t take too much from them.

ImDisk does sound like it can do persistent disk by saving, but what if it cannot? Would it tell you on next start that it only has an old version of the data around?

I don’t use the ramdrive with persistent backing. I’m not copying the backup DB to the ramdrive. Just using the ramdrive for temp storage, exactly as @DCinME describes below with the tempdir option. Sorry if that was unclear from my OP.

Regarding the filesystem, I definitely had it configured to NTFS. My intitial thought is there’s no way I’d use FAT … but then, perhaps that’s not a bad point. The ramdrive was used solely by Duplicati so perhaps NTFS was not only overkill but getting in the way. Worth testing. Thanks.

8

Thanks for sharing.

Yeah, that’s how I was using it, via the tempdir option.

The fact that you’ve used the same setup for some time gives me confidence that I wasn’t doing something stoopid (yet to be conclusively ruled out though) and that it’s viable.

I’m sure I saw SQLite temp files (eg etilqs*) in the ramdrive while I was using.it. Pretty sure I wasn’t using %TEMP% to redirect to the ramdrive because I initially tried that and it didn’t work, but maybe it’s in a script and I forgot. I also remember searching for any environment variables specific to SQLite that could be used though didn’t find anything. Kenkendk suggests the SQLITE_TMPDIR variable to prevent SQLite using the tempdir location.

Anyway, I appreciate the information, regardless.

9

During the normal course of using Duplicati I don’t believe I see etilqs files in the in the RAM drive, however I have seen them there during other operations that involve creating temporary copies of the database.

For example, every now and then I do restoration tests where I restore directly from the remote backup to make sure that the backup is usable even if my local database and files are lost. During these operations Duplicati does put the temporary copy of the database in the RAM drive, but I can’t think of another time it has done that.

10

FAT32 doesn’t support file locking, so is probably worse if that’s the issue.

11

Re the previous forum posts from 2018 you mentioned that I blithely dismissed in my reply above … yeah, I’m a shmuck. Lots of good info there.

Thanks.

12

Duplicati tries to “fix” this:

Note that the if is not yet in any canary; previous versions would unconditionally set all of them on load.

13

I’m not sure which part is attempting a fix, but if it’s to move SQLite temporary files:

Temporary File Storage Locations (at sqlite.org) describes limits on fix attempt.

Is there a path seen there that lets Windows do it by setting environment variables?

They did name PRAGMA temp_store_directory, but docs for it suggest not using it.

14

Yes, I meant the SQLite specific temp folder location.

It is quite a mess from SQLite’s side with no clear recommendation for how to do it, so Duplicati can only try and hope for the best.