Problem backing up data from USB stick

Dear all,

please excuse me, probably this is not a problem caused by Duplicati but maybe someone has an advice for me …

I have an encrypted VeraCrypt volume (180 GB) on an USB stick. This volume is mounted as a drive and I want to back up the data inside (170 GB, about 200k files). Duplicati starts the backup successfully, but after some time (1 hour, sometimes 3 hours) the USB stick is automatically unmounted (Windows sound like unplugging the stick) which of course causes the backup to fail.

My suspicion is that due to the massive (parallel?) access to the files on the USB stick, either some driver (VeraCrypt driver? USB driver?) crashes or the USB stick’s power consumption exceeds what the computer can provide.

A backup of only a part of the files (about 8 GB) runs fine. I have also tried to change the USB port without any success.

I have found an option to throttle the bandwith but only for uploading to the destination, I have not found any option to reduce the read access to the source files.

Does anyone of you have an advice for me?

Thank you and greetings,
Chris

Hi @cgaertner!

Welcome to the forum.

Have you looked at the Windows event logs to see what it is reporting around that time? I am sure it will say something if even to indicate that the device was “lost”. But maybe there’s some additional information that will help.

Regards,
L

Hi @Lerrissirrel,

thank you for your answer. This was a good idea, I run another new full backup this afternoon.

It started at 16:22:05 and run until 18:31:54 (02:09:49 hours) and backed up successfully 135,771 files (84.68 GB). Then suddenly the USB stick was ejected. The event log only shows three entries in the relevant time frame:

18:29:34 hcmon Detected unrecognized USB driver (\Driver\pci).
18:29:34 hcmon Detected unrecognized USB driver (\Driver\ACPI).
18:29:34 hcmon Detected unrecognized USB driver (\Driver\UDE).

I will try to investigate; maybe this is this something someone here knows more about.

Thank you!

Greetings, Chris

I was going to suggest a potential power management aspect before, but decided not to. However, seeing the ACPI driver mentioned here, it is probably worth investigating whether some power management setting is disabling power to your USB port. I would not expect Windows to power the USB down if it was actively in use, but I don’t know how Windows power management determines “idle”.

Long story short - you could try playing around with the advanced power management settings.

L

I just tried to re-run the previously interrupted backup and this time it failed twice after a few minutes with the same error.

By the way: there is no problem if I copy all the files with RoboCopy, so I am sure that it is no read-error or something similar on the USB stick.

I will investigate further regarding the power management.

Thanks, Chris

Google search for this sort of report found several, but not many with solutions that worked.
Volume auto-dismounts at seemingly random times was one of them, and I notice that it’s in VeraCrypt’s forum where you might be able to find others, or if not, maybe report symptoms.

  --concurrency-block-hashers (Integer): Specify the number of concurrent
    hashing processes
    Use this option to set the number of processes that perform hashing of
    data.
    * default value: 2

Reducing to 1 might be able to make the drive slightly less busy, but it’s doing other things too.

Channel Pipeline describes, but even GUI will show it’s counting files while also reading files…

  --use-background-io-priority = false
    to use the lowest IO priority level, which can make operations run slower

might slow it by yielding to other I/O, but is there any other I/O active that would make it do so?

If you have or can acquire other USB devices, you could see whether this seems like the cause.

Thank you, @ts678 for your input!

One thing that I maybe did not point out clear: the VeraCrypt volume is not unmounted, it is the USB stick that is “ejected”! The VeraCrypt volume is still present, of course data can no longer be read as the USB stick is gone. BTW: it is painful to unmount a VeraCrypt volume when the USB stick is missing. In fact I just physically plugged out and in the USB stick and then the unmounting process works fine.

Your hint with the “concurrency-block-hashers” option is worth a try, I noticed that the default value in my installation is 8! I also noticed another option, “concurrency-compressors”, which has also a default value of 8. I will set these values to 1, use background priority and reduce the number of threads to 1. I will start another backup job with these settings and give feedback here. But unfortunately I will not be able to do this before Monday.

Greetings, Chris

PS: What I also did: I mounted the VeraCrypt volume “read-only” to (1) prevent data loss caused by the USB device removement and (2) to prevent the NTFS file system to set the “last access” timestamp to every file that is read (?).

PPS: All power saving options are turned off for the USB device. I am using this device with the mounted VeraCrypt volume every day for 8+ hours and it never gets ejected. Thus my suspicion that it has to do with the massive file access …

It’s going to access files, no doubt about it. I don’t know if read rate matters here, but you can set

  --blocksize (Size): Block size used in hashing
    The block size determines how files are fragmented. Choosing a large value
    will cause a larger overhead on file changes, choosing a small value will
    cause a large overhead on storage of file lists. Note that the value
    cannot be changed after remote files are created.
    * default value: 100kb

up as high as the Options screen Remote volume size (and thereby hurt deduplication) for a test.

You could also try to separate the file counting load from the file reading load by excluding all files

Use a test job for these of course, and of course the test just above will get info but not a backup.

Do you know which ports if any are USB 3? I think they do 900 mA compared to USB 2 500 mA.

HWiNFO64 seems to be one way to see. It also shows which USB the stick is – or check specs.

If your device needs USB 3 level power, and you have no such ports, a powered hub might help.

USB volt/amp meters also exist if you want to see if USB power draw grows too high during use.

Thank you for the replies!

A short update what I found out in the meantime:

(1) I copied the whole 180 GB Veracrypt volume to my harddisk, mounted it and did a full backup. As I expected, it worked without any problems. This confirms that there is no problem with the Veracrypt driver. I estimated this to be very unlikely, but now it’s a point I can cross of the list.

(2) I copied the whole 180 GB Veracrypt volume to an external USB harddisk and did a full backup, which also worked without any problems. I actually used the default settings (block size, threads, …) so it run at full speed. This external harddisk is an USB 2 device, which obviously needs less current than my USB stick.

I did play around with HWiNFO64 which gave me detailed information about the external harddisk (USB 2) and the USB stick (USB 3), but I could not find out which USB port can deliver how much current. I will have to investigate further. And I also want to try to get a powered USB hub to check if this makes any difference.

Greetings, Chris

I think it’s implicit by knowing whether it’s USB 2 or USB 3 (might need to read the names).

In this case both USB ports that I have tried are USB 3 ports.

Some news from my side:

In the meantime I purchased a powered USB hub and connected the USB stick to this hub. The backup was running last night, it took about 5 hours (about 10 MB/sec) and completed without any problems! :slight_smile:

For me this leads to one of these two conclusions:

(1) The built-in USB ports deliver not enough current for the USB stick during the massive amount of read access running a several hour 180 GB backup job.

(2) The USB stick is somehow defective. I think this is unlikely because it works on several other computers and even a copy operation with RoboCopy of the whole 180 GB VeraCrypt image runs fine. On the other hand this is also an argument against assumption (1) above. :thinking:

I think I will install Duplicati on my work computer next week and try to run a backup there. Same USB stick, other computer. I will report.

Greetings, Chris