@ts678, I was going through your last post trying everything one recommendation at a time and when I got to explicitly setting the HTTP operation timeout I ran into this weirdness. Do you know why I have two of every option and f there is a difference between the two?
OneDrive (personal) back-ups suddenly started failing with "Channel is retired" and "A task has been cancelled"
The issue with duplicate http options should have been fixed in version 220.127.116.11. What version are you running?
TaskCancelled means that “something” has requested that the uploader process should stop, and not attempt to upload any more files, which is why it fails to retry.
Channel is retired is essentially the same message; “something” has stopped working and is shutting everything down.
I am not sure why it does not work, but in this case there should be multiple error messages that explain the root cause of the failure, instead of just the useless error messages you see.
Could it be something with your quota on OneDrive being exceeded suddenly?
I am running 18.104.22.168_beta_2018-11-28, which as far as I am aware is the most recent version on the beta update channel. Considering that this is the most stable channel and the warnings on the other two channels, I would be reluctant to update.
Do you know if that issue is just a UI defect that does not affect functionality (i.e. both options work)?
I wouldn’t say I have multiple error messages, but based on your statement, I will go back to the log file and look again - maybe I missed it.
Not very likely - my quota is 1TB and it is less than 50% full, so it should be able to take at least the first few dblocks without any issues, but it fails instead. Also the overall size of the upload, should be waaay less than the free space because it is essentialy only the delta since the last back-up. Unless Duplicati is check the total size of the back-up against the available free space (even though it will only need to upload a fraction of that). Also, the quota theory doesn’t explain why it would fail on my test back-up - its total size is around 100MiB and I still have hundreds of gigs available in OneDrive.
I still haven’t had a chance to test this all the ways suggested by ts678, but I think I should be able to do it later today or tomorrow.
Thanks for all the replies and suggestions!!!
The duplicated http options was just a bug in the UI. Both options should work.
Thanks for clarifying!
@ts678 and everyone else who might know - I have been experimenting with these options for the last couple of days and while –number-of-retries did not seem to have any effect, I was able to make some progress with –http-operation-timeout. Considering that I am getting various failures mixed with some successes, I would like to ask:
- What is a reasonable value for the http timeout?
- Does the value depend on my connection speed and bandwidth (i.e. poor connections works better with a higher value and vice versa)? Any real world examples?
The value you need depends entirely on how long your network needs to transfer your file. Slow networks need longer timeouts. You can see how long your files are actually taking by watching About --> Show log --> Live --> Profiling and looking at the Uploading and Downloading lines for size and time (more size --> more time). Adding to the job the options --log-file and --log-file-log-level=Profiling will work too. Examples:
2018-11-28 08:22:42 -05 - [Profiling-Duplicati.Library.Main.BackendManager-UploadSpeed]: Uploaded 50.02 MB in 00:01:42.3183110, 500.63 KB/s
2018-11-26 15:57:53 -05 - [Profiling-Duplicati.Library.Main.BackendManager-DownloadSpeed]: Downloaded and decrypted 49.99 MB in 00:00:12.2495337, 4.08 MB/s
Upload speed is lower than download on many Internet connections. You can speed test your connection. Units may use multiples of bits (lowercase b), or bytes (uppercase B, where 1 byte = 8 bits). Division will give the theoretical minimum time, but other factors (such as CPU, disk, Duplicati, etc.) may slow things.
If you have set your job screen 5
Remote volume size larger than 50MB default, that will also take longer. Basically you want to leave enough time for your slowest transfer on slowest network with some margin…
Here are notes on --http-transfer-time, and OneDrive success using smaller volumes and larger timeouts.
This is AWESOME!!!
Thank you very much for the detailed explanation!
I guess this is my problem… It still remains a mystery why this is NOT an issue with AWS S3 (I still have a couple of back-ups on S3) and why it worked fine the first few days after I switched to OneDrive.
I will try tweaking the volume size and the timeout further and hopefully that will resolve the issue.
On another note, there is clearly an opportunity to improve Duplicati’s error logging and reporting facilities - a problem like this is rather trivial and should be clearly indicated in the error messages on screen and in the log. Duplicati already does this in some of the other error messages that not only state exactly what the problem is, but also have a recommendation about solving it.
AWS S3 uses Amazon’s AWSSDK*.dll instead of direct .NET Framework access. You can see the .NET timeout config reflected in theirs. Their default looks undocumented, but code might set it extremely high, which can prevent prompt enough recognition of problems, resulting in apparent hangs. Discussion here.
No argument on the general statement. It frustrates users and also those who have to point to all the logs.
however I’m not sure I agree with this. It took a lot of tracking down and experimentation by us to resolve it (and we’re not even sure yet that it is resolved). As said earlier, “Channel is retired” is a very generic error.
There’s a statement from the lead developer somewhere hoping for pretty much what you suggest at end, however there are many competing priorities. There are also statements that logging is messy – and hard.
I ran into the same issue recently and thus second that very question.
I am no developer, and I apologize if I have underestimated the complexity there. My point was that a connection timeout should be pretty straightforward to detect and report to the user, but to your point the underlying .NET may not necessarily make that easy.
Not sure if you mean our discussion above until we sighted in on the timeout, but if you do - I agree and I appreciate it very much.
Correct. I seem to be making progress, however - it is a huge back-up (280 GiB + whatever has changed since it broke) and was running quite well until some time today. I was monitoring it and I saw that with a remote volume size of 25 MiB and a 6 min. timeout it was barely making it (dblock files would finish uploading just before hitting the 6 min. mark). Eventually it failed.
I am retrying with a 20 MiB remote volume size and a 7 min. timeout, but I haven’t been able to gauge how well it’s going.
You’re getting perhaps 350 kilobits/second. Can you find an Internet speed test to see if it matches, and then see what your ISP is promising, and whether they can help? In my home, I sometimes have to reconnect my PC to Wi-Fi or restart my router or (less often) restart cable modem, and occasionally get ISP service visits.
Sometimes speeds vary with time of day (slow at busy times), packet loss slows things (try a ping test), etc. Gauging approximate network performance for the connection can be done on Windows with Task Manager.
I wish! I use a relatively cheep service and my upload speed is lower, both as promised by the ISP and in reality.
That is not the problem though - Duplicati was perfectly happy with that before switching from AWS to OneDrive and even on OneDrive before upgrading and switching to OneDrive v2. But we already discussed some of that. Plus my set-up is far from simple, so there are other factors at play as well
Anyway! I am happy to report that with an 8 min. timeout and a remote volume size of 20 MiB, I finally have my first successful real back-up for more than a month!
Thank you to everyone who responded and especially to @ts678!
The second question (the one you quoted) was answered indeed, but not the first one.
To make this clear: In order to get my Azure backup working again, I need to switch to an unstable update channel or wait until 22.214.171.124 is available for beta?
What’s wrong with your backup? I see no information anywhere. Have you followed the steps here to see whether you’re getting timeouts? If so, then somebody (perhaps me) can then go off to see how one sets larger timeouts for Azure. One challenge with timeouts is that different destination type may set differently. For example, setting an http timeout on SFTP or FTP wouldn’t make any sense. At least Azure uses http, however (just as AWS does) it has its own application programming interface (which I’m not familiar with).
Release: 126.96.36.199 (canary) 2018-12-11 is already out, and it’s canary. Numbers are not meant to be reused. If you’re just asking if you need to wait for next beta, that’s generally only an option for a fix at all if a canary or experimental (less bleeding edge than canary) has the fix. Not knowing the issue, I can’t say anything…
Well the issue I suppose is exactly the one @rumenavramov was reporting initially: I’ve had a OneDrive backup working for over a year - albeit in rather irregular schedules, the last successful one dating from December. When I wanted to let it run again recently, I first ran into the issue that apparently the OneDrive API was deprecated and needed to be changed to v2. Afterwards I got that very “channel is retired” log entry.
When I read now that this issue has been fixed in a version released two months ago, my primary question (rather than digging deep into troubleshooting) is how do I get that version without “breaking anything” (i.e. having a stable version that includes this fix). The OneDrive backup is only my 2nd copy and as I’ve said I didn’t stick to a strict schedule, so I’m well able to wait rather than switching to the canary channel that gets me a couple of other issues that need to be fixed again…
I think that might have been a misreading, but feel free to point or quote. Meanwhile, this is my view of it.
The only two months ago fix I see is the above 188.8.131.52 (Dec 11 release) one where the duplicated options were fixed by Remove duplicate connection entries from advanced options list #3511, however you rightly seem more interested in a fix for a backup failure. v184.108.40.206-220.127.116.11_canary_2018-12-11 is another change log view which is part of the series if you want to look it over yourself to see what fixes are being released. This can give you an idea of how often beta happens, and you can also see how fixes go into canary first.
Additionally, as well-covered here, “Channel is retired” means little by itself, and needs one to look for the original error. We don’t even know for sure you got timeouts. You can check as shown here. Alternatively just try the posted solution. If increasing the timeout fixes the problem, you had timeouts. You also might just know if you have unusually slow Internet, or custom settings on things like large remote volume size.
Anything other than timeouts probably deserves a new topic, where it can (hopefully) get its own solution. There is no way (to my knowledge) for a single forum post to have multiple solutions presented at the top.
It’s too bad original OneDrive is shut down at Microsoft. There’s no way to do head-to-head comparisons.