please excuse me, probably this is not a problem caused by Duplicati but maybe someone has an advice for me …
I have an encrypted VeraCrypt volume (180 GB) on an USB stick. This volume is mounted as a drive and I want to back up the data inside (170 GB, about 200k files). Duplicati starts the backup successfully, but after some time (1 hour, sometimes 3 hours) the USB stick is automatically unmounted (Windows sound like unplugging the stick) which of course causes the backup to fail.
My suspicion is that due to the massive (parallel?) access to the files on the USB stick, either some driver (VeraCrypt driver? USB driver?) crashes or the USB stick’s power consumption exceeds what the computer can provide.
A backup of only a part of the files (about 8 GB) runs fine. I have also tried to change the USB port without any success.
I have found an option to throttle the bandwith but only for uploading to the destination, I have not found any option to reduce the read access to the source files.
Have you looked at the Windows event logs to see what it is reporting around that time? I am sure it will say something if even to indicate that the device was “lost”. But maybe there’s some additional information that will help.
thank you for your answer. This was a good idea, I run another new full backup this afternoon.
It started at 16:22:05 and run until 18:31:54 (02:09:49 hours) and backed up successfully 135,771 files (84.68 GB). Then suddenly the USB stick was ejected. The event log only shows three entries in the relevant time frame:
18:29:34 hcmon Detected unrecognized USB driver (\Driver\pci).
18:29:34 hcmon Detected unrecognized USB driver (\Driver\ACPI).
18:29:34 hcmon Detected unrecognized USB driver (\Driver\UDE).
I will try to investigate; maybe this is this something someone here knows more about.
I was going to suggest a potential power management aspect before, but decided not to. However, seeing the ACPI driver mentioned here, it is probably worth investigating whether some power management setting is disabling power to your USB port. I would not expect Windows to power the USB down if it was actively in use, but I don’t know how Windows power management determines “idle”.
Long story short - you could try playing around with the advanced power management settings.
Google search for this sort of report found several, but not many with solutions that worked. Volume auto-dismounts at seemingly random times was one of them, and I notice that it’s in VeraCrypt’s forum where you might be able to find others, or if not, maybe report symptoms.
--concurrency-block-hashers (Integer): Specify the number of concurrent
hashing processes
Use this option to set the number of processes that perform hashing of
data.
* default value: 2
Reducing to 1 might be able to make the drive slightly less busy, but it’s doing other things too.
Channel Pipeline describes, but even GUI will show it’s counting files while also reading files…
--use-background-io-priority = false
to use the lowest IO priority level, which can make operations run slower
might slow it by yielding to other I/O, but is there any other I/O active that would make it do so?
If you have or can acquire other USB devices, you could see whether this seems like the cause.
One thing that I maybe did not point out clear: the VeraCrypt volume is not unmounted, it is the USB stick that is “ejected”! The VeraCrypt volume is still present, of course data can no longer be read as the USB stick is gone. BTW: it is painful to unmount a VeraCrypt volume when the USB stick is missing. In fact I just physically plugged out and in the USB stick and then the unmounting process works fine.
Your hint with the “concurrency-block-hashers” option is worth a try, I noticed that the default value in my installation is 8! I also noticed another option, “concurrency-compressors”, which has also a default value of 8. I will set these values to 1, use background priority and reduce the number of threads to 1. I will start another backup job with these settings and give feedback here. But unfortunately I will not be able to do this before Monday.
PS: What I also did: I mounted the VeraCrypt volume “read-only” to (1) prevent data loss caused by the USB device removement and (2) to prevent the NTFS file system to set the “last access” timestamp to every file that is read (?).
PPS: All power saving options are turned off for the USB device. I am using this device with the mounted VeraCrypt volume every day for 8+ hours and it never gets ejected. Thus my suspicion that it has to do with the massive file access …
It’s going to access files, no doubt about it. I don’t know if read rate matters here, but you can set
--blocksize (Size): Block size used in hashing
The block size determines how files are fragmented. Choosing a large value
will cause a larger overhead on file changes, choosing a small value will
cause a large overhead on storage of file lists. Note that the value
cannot be changed after remote files are created.
* default value: 100kb
up as high as the Options screen Remote volume size (and thereby hurt deduplication) for a test.
You could also try to separate the file counting load from the file reading load by excluding all files
(1) I copied the whole 180 GB Veracrypt volume to my harddisk, mounted it and did a full backup. As I expected, it worked without any problems. This confirms that there is no problem with the Veracrypt driver. I estimated this to be very unlikely, but now it’s a point I can cross of the list.
(2) I copied the whole 180 GB Veracrypt volume to an external USB harddisk and did a full backup, which also worked without any problems. I actually used the default settings (block size, threads, …) so it run at full speed. This external harddisk is an USB 2 device, which obviously needs less current than my USB stick.
I did play around with HWiNFO64 which gave me detailed information about the external harddisk (USB 2) and the USB stick (USB 3), but I could not find out which USB port can deliver how much current. I will have to investigate further. And I also want to try to get a powered USB hub to check if this makes any difference.
In the meantime I purchased a powered USB hub and connected the USB stick to this hub. The backup was running last night, it took about 5 hours (about 10 MB/sec) and completed without any problems!
For me this leads to one of these two conclusions:
(1) The built-in USB ports deliver not enough current for the USB stick during the massive amount of read access running a several hour 180 GB backup job.
(2) The USB stick is somehow defective. I think this is unlikely because it works on several other computers and even a copy operation with RoboCopy of the whole 180 GB VeraCrypt image runs fine. On the other hand this is also an argument against assumption (1) above.
I think I will install Duplicati on my work computer next week and try to run a backup there. Same USB stick, other computer. I will report.