Another backup failed due to error Cannot find file

Plain FTP protocol (i.e. not FTPS) doesn’t encrypt username and password, so be careful.

attempt 1 of 5 failed with message: HTTP timeout 00:01:40 exceeded.

is likely the default Microsoft 100 second timeout (although the log shows long ones working).
–http-operation-timeout raising can generally get rid of these if the upload needs more time…
I’ve suggested increasing it (because this may be the only one so short), but it’s not done yet.

It’s also my suspicion but I haven’t been able to invent a test case that breaks in just that way.
I’ve seen some other ways for it to break though… This seems a target area for robustness…

Thanks for adding a log to the collection. Maybe that will lead to added insights on the error…

So I’ve set http-operation-timeout to 5 minutes and thus far the backup is running steady.

However, it’s using all my uplink bandwidth and when I try to limit that to 1/5th, it does not have any effect. Since we all need to work and learn from home because of Corona, that’s causing bandwidth issues for e.g. Skype for Business. Why is it ignoring the max. up- and download speed settings?

Be careful about units. If you have (say) a 10Mbit/sec upload and set Duplicati to 2 MByte/sec, you’ve requested it to use up to 16 Mbit/sec because 1 byte is 8 bits. Note that there is a bug fix in next Beta which will unconfuse --throttle-upload and --throttle-download. 2.0.5.1 upload setting may throttle both.

I’m aware of that, so I’m using the same unit (only KB/s) as displayed in the progress bar and set the speed limit to 1/5th of that, but with no affect.

Still, at least the backup has been running steadily now for over 3 days or so!

Did you change it before the backup job started? From what I recall you cannot change the speed of a job already in progress.

Although not 100% of the destination types support throttling, OneDrive v2 does if that’s what’s in use.

Here’s what happens on mine without throttling (at 11:20) and with --throttle-upload=100KB (at 12:20):

2020-04-27 11:20:34 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 45.25 KB in 00:00:00.8894220, 50.87 KB/s
2020-04-27 11:21:23 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.63 MB in 00:00:50.1186602, 482.89 KB/s
2020-04-27 11:21:26 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 35.20 KB in 00:00:03.3229027, 10.59 KB/s
2020-04-27 12:20:40 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 45.29 KB in 00:00:01.8628526, 24.31 KB/s
2020-04-27 12:28:02 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 35.96 MB in 00:07:24.5803222, 82.83 KB/s
2020-04-27 12:28:04 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 41.14 KB in 00:00:01.9487112, 21.11 KB/s

Duplicati 2.0.5.1 by default has added a concurrent upload capability in order to increase the speed.
I’m guessing that’s why my 100 KByte/s setting doesn’t quite get hit, or it might be something else…

  --asynchronous-concurrent-upload-limit (Integer): The number of concurrent
    uploads allowed
    When performing asynchronous uploads, the maximum number of concurrent
    uploads allowed. Set to zero to disable the limit.
    * default value: 4

A similar test to Google Drive with that dialed back to 1 concurrent was just about at target 100KB/s:

2020-04-27 10:48:01 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 42.47 MB in 00:07:22.9627872, 98.17 KB/s

I stopped the backup and restarted it, but that doesn’t make a difference. Duplicati keeps using all the bandwidth it can get its hand on.

You’ve tried several destination types. Is the current one below?

image

Are your speed results strictly from the Duplicati status bar display? What does Task Manager → Performance show for whatever device is doing the uploads? View of an upload is from the logs.
–log-file can be set in Advanced options, and instead of level you can use –log-file-log-filter of Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed

Whatever you’re seeing, it’s disagreeing with my posted OneDrive results showing OK throttling.
Although I’m not sure they interact, you might also check the speed control at top of home page:

image

EDIT:

Although my prior tests concluded that upload throttle works in 2.0.5.1, I’m currently on 2.0.5.104, so setting –throttle-download could be tried in addition. You could also try the GUI speed control setting.

image

Yes it is.

Yes they are. I figured that staying in the Duplicati “domain” gave the best chance of having matching numbers.

It shows approximately 5 Mb/s whereas Duplicati shows approximately 500 KB/s, so that matches.

I’m sorry, but I’m not sure what you want me to do here.

That’s the one I used as it is right next to the speedometer. Please have a look at the attached screenshot:

It clearly shows that the upload speed is over 500 KB/s while the Max upload speed is set to 100 KByte/s. That’s weird, right?

Best regards,
Patrick.

The other way of setting is on Options screen 5 in Advanced options. I don’t know which one “wins”:

The log setup is also in Advanced options if you want to see what the speed is over each file upload. Generally my experience is that short files don’t reach full speed, but larger ones (like dblock) can do.
Of course the problem you seem to be seeing is that it goes beyond what you consider the speed limit.

Anyway, the per-file detailed log file look I had in mind is to pick and enter path to receive –log-file and

to get a limited amount of output rather than the flood that --log-file-log-level=profiling would give you.
The flood has advantages, e.g. you see the file names. I suppose you can try About --> Show log --> Live --> Profiling to see if any speeds can be picked out from all of the other activity (e.g. SQL action).

Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed
is the content of the entry box (whose rightmost portion is hidden) as taken from my log example above which showed my throttle value actually being used. I’m not sure why yours is seemingly being ignored, however the detailed log may give a better view of the overspeed. I could possibly also test with 2.0.5.1.

Weird, yes. I tested 2.0.5.1 with a variety of combinations of the GUI control set as yours, along with --throttle-upload in either per-job or global settings. Going to OneDrive v2, it seemed like the throttle worked at whichever was less between the GUI control and the job setting. Here are the test results:

With 100 KByte/s on GUI Throttle settings, and no other throttle, throttle seems to respect setting:

2020-04-29 15:55:11 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 701 bytes in 00:00:00.5594441, 1.22 KB/s
2020-04-29 15:55:49 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.55 MB in 00:00:38.8796512, 93.49 KB/s
2020-04-29 15:55:49 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.11 KB in 00:00:00.3248541, 71.13 KB/s
2020-04-29 15:57:20 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:10.3090614, 78.41 KB/s
2020-04-29 15:57:21 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.96 KB in 00:00:00.9152339, 4.33 KB/s
2020-04-29 15:57:21 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:12.1042411, 77.35 KB/s
2020-04-29 15:57:22 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 4.00 KB in 00:00:00.5800420, 6.89 KB/s

With 100 KByte/s on GUI Throttle settings, and --throttle-upload=50 KByte/s on job, job option wins:

2020-04-29 16:02:07 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 699 bytes in 00:00:00.5599073, 1.22 KB/s
2020-04-29 16:03:08 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 2.87 MB in 00:01:01.3698865, 47.85 KB/s
2020-04-29 16:03:10 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.10 KB in 00:00:01.5899313, 14.53 KB/s
2020-04-29 16:05:59 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:03:52.6132017, 43.93 KB/s
2020-04-29 16:06:00 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.95 KB in 00:00:00.6201311, 6.37 KB/s
2020-04-29 16:06:02 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:03:55.2081937, 43.44 KB/s
2020-04-29 16:06:02 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 4.00 KB in 00:00:00.4500265, 8.88 KB/s

With 100 KByte/s on GUI Throttle settings, and --throttle-upload=200 KByte/s on Settings --> Default options, GUI Throttle settings wins:

2020-04-29 16:12:55 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 701 bytes in 00:00:00.5359672, 1.28 KB/s
2020-04-29 16:13:11 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 1.41 MB in 00:00:16.3897179, 87.86 KB/s
2020-04-29 16:13:11 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.11 KB in 00:00:00.3099663, 74.55 KB/s
2020-04-29 16:15:01 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:06.1993172, 80.97 KB/s
2020-04-29 16:15:01 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 4.00 KB in 00:00:00.5898743, 6.78 KB/s
2020-04-29 16:15:07 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:12.3092231, 77.23 KB/s
2020-04-29 16:15:07 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.96 KB in 00:00:00.4650703, 8.51 KB/s

With 100 KByte/s on GUI Throttle settings, and --throttle-upload=50 KByte/s on Settings --> Default options, Settings wins.

2020-04-29 16:25:40 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 700 bytes in 00:00:00.6300016, 1.08 KB/s
2020-04-29 16:26:06 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 1.21 MB in 00:00:26.9725838, 45.99 KB/s
2020-04-29 16:26:08 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.11 KB in 00:00:01.6071419, 14.38 KB/s
2020-04-29 16:29:48 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:04:09.2910697, 40.99 KB/s
2020-04-29 16:29:49 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.99 KB in 00:00:00.7648334, 5.22 KB/s
2020-04-29 16:29:55 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:04:15.7155883, 39.96 KB/s
2020-04-29 16:29:55 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.96 KB in 00:00:00.6494684, 6.09 KB/s

With 100 KByte/s on GUI Throttle settings, and --throttle-upload=200 KByte/s on job, GUI Throttle settings wins:

2020-04-29 16:35:34 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 701 bytes in 00:00:00.6902640, 1015 bytes/s
2020-04-29 16:35:52 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 1.72 MB in 00:00:19.5198508, 90.34 KB/s
2020-04-29 16:35:53 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.11 KB in 00:00:00.4399550, 52.52 KB/s
2020-04-29 16:37:58 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:25.6295808, 70.16 KB/s
2020-04-29 16:37:59 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.95 KB in 00:00:00.5603032, 7.06 KB/s
2020-04-29 16:38:00 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:02:27.2639085, 69.39 KB/s
2020-04-29 16:38:00 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.99 KB in 00:00:00.4852013, 8.23 KB/s

This used a 10 MByte Remote volume size on Options screen 5 because source file was a bit small.
Beyond possible --throttle-upload per test note, the only Advanced options were for setting up log file.

I changed the asynchronous-concurrent-upload-limit to 1 (as also mentioned here as a solution. This solved the problem in my case. The max upload speed is now respected and my back-up is running steady for 5 hours. With the default value set to 4, I saw heigh peaks in the upload speed and it was way above the set limit.

Thanks for the input @Niels

I’m glad that helped. My results were good without it, and with it was also good (below 100 KBytes/s).

2020-04-30 17:47:12 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:01:57.8697837, 86.69 KB/s
2020-04-30 17:47:13 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 4.01 KB in 00:00:00.6301099, 6.36 KB/s
2020-04-30 17:49:11 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 9.98 MB in 00:01:57.7370898, 86.79 KB/s
2020-04-30 17:49:11 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 3.96 KB in 00:00:00.5905629, 6.70 KB/s
2020-04-30 17:50:14 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 5.34 MB in 00:01:02.5895033, 87.34 KB/s
2020-04-30 17:50:14 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 23.11 KB in 00:00:00.4000050, 57.77 KB/s
2020-04-30 17:50:15 -04 - [Profiling-Duplicati.Library.Main.Operation.Backup.BackendUploader-UploadSpeed]: Uploaded 701 bytes in 00:00:00.2797670, 2.45 KB/s

I think the algorithm has always been aiming at an average rate (so peaks can happen), and possibly peaks can be larger running concurrent uploads. I think downloads are more peaky because Duplicati doesn’t have control of the remote sender rate. On uploads, it can dribble data out to limit upload rate.

For best speed control, something done at the router (if it’s available) can likely do better than an app.