OnedriveV2 not available as provider choice

ts678 · September 9, 2018, 12:14pm

I think this arrived in Release: 2.0.3.4 (canary) 2018-04-02. Pull request: Add new backends for OneDrive + OneDrive for Business, SharePoint, and Microsoft Groups based on the Microsoft Graph API #3118

EDIT: I’m not sure what I saw before, but I transcribed 2.0.3.3 beta and 2.0.3.11 canary Storage Type values:

2.0.3.3 Beta
Microsoft OneDrive
Microsoft OneDrive for Business
Microsoft SharePoint

2.0.3.11 Canary
Microsoft Office 365 Group
Microsoft OneDrive
Microsoft OneDrive for Business
Microsoft OneDrive v2
Microsoft SharePoint
Microsoft Sharepoint v2

I’m concerned about timing, due to Microsoft’s deadline, and this feature only being in canary release so far. Possibly migration aids and/or publicity is advisable in order to reduce broken backups come November 1st. Doing another Beta has been a desire, but I’m not sure when it will happen or if an Experimental would help.

Old OneDrive API being deprecated - needs to be rewritten to use the Graph API #3024

Now back to the actual latest question , I don’t run Docker but I’m wondering if psywolf85 runs CLI or web.

$ duplicati-cli help onedrivev2
Microsoft OneDrive v2 (onedrivev2): <etc.>`

is one way to see if at least that code knows onedrivev2. Also check web About --> System info version info, consider possibility of browser needing a page refresh, cache clear, restart, or whatever (maybe view is old).

anon77588929 · September 9, 2018, 1:44pm

Hi, I am facing the same issue as psywolf85 with the official Docker images.

This issue has been reported to Github previously.

https://github.com/duplicati/duplicati/issues/3198

root@duplicati-duplicati1:/# duplicati-cli help onedrivev2                                                         
Topic not found: onedrivev2                                                                                        


See duplicati.commandline.exe help <topic> for more information.
(truncated)

ts678 · November 11, 2018, 11:52pm

Official docker image is not including all backends #3198 is my update with a lot of technical detail, but mostly being able to characterize the problem would be a good start. @psywolf85 almost sounds like it’s not only a OneDrive v2 problem, but a OneDrive of all sorts problem, which would be what I (rather disappointingly) saw:

Microsoft Office 365 Group
Microsoft OneDrive
Microsoft OneDrive v2
Microsoft SharePoint v2

were all missing. How does this compare to your lists? Are you missing more? Less (got Microsoft OneDrive)? Feedback from @anon77588929 and others with storage types missing in official Docker would aid understanding. Don’t look for it in 2.0.3.3, and ideally see how the new Release: 2.0.4.1 (experimental) 2018-11-08 does on a test system (because I’m not certain what database formats changed when, and downgrading can be difficult).

JonMikelV · November 12, 2018, 2:49am

I compared 2.0.3.14 canary installs between Windows 10 (updated from 2.0.2.1 beta) and the official Docker container (updated from 2.0.3.3. beta) and they are the same EXCEPT that the items you listed are indeed missing from the Docker container:

Microsoft Office 365 Group
Microsoft OneDrive
Microsoft OneDrive v2
Microsoft SharePoint v2

Note that 2.0.4.1 experimental in both the official AND linuxserver Docker containers appears to have the complete list (so DO include the above 4 Microsoft items).

ts678 · November 12, 2018, 3:24am

I’m starting with a new one: duplicati/duplicati:linux-amd64-2.0.4.1_experimental_2018-11-08 without attempting any updates to an existing container, if that’s what you’re doing. I also tried a new 2.0.3.4.

??? Does that mean these are two different containers? I consider the one listed at top as “official Duplicati”. Regardless, you’re seeing this oddity at least sometimes, and the unpredictability of results does bother me.

From a shell inside container, this tends to find onedrive (unlike web UI) but no v2:

# pwd
/opt/duplicati
# head -1 changelog.txt
2018-11-08 - 2.0.4.1_experimental_2018-11-08
# mono Duplicati.CommandLine.BackendTool.exe help
Usage: <command> <protocol>://<username>:<password>@<path> [filename]
Example: LIST ftp://user:pass@server/folder

Supported backends: aftp,amzcd,azure,b2,box,cloudfiles,dropbox,ftp,file,googledrive,gcs,hubic,jottacloud,mega,onedrive,openstack,rclone,s3,ssh,od4b,mssp,sia,tahoe,webdav
Supported commands: GET PUT LIST DELETE CREATEFOLDER
#

JonMikelV · November 12, 2018, 3:39am

So far I’ve seen this oddity only BETWEEN VERSIONS but not “sometimes yes sometimes no” within the same version. (EDIT: SEE BELOW)

My guess is there was a mistake with the automated build script for the Docker containers that has been resolved as of 2.0.4.1 experimental.

EDIT:

Yes - though there was a typo in my post. I meant to say the official (https://hub.docker.com/r/duplicati/duplicati/) and linuxserver (https://hub.docker.com/r/linuxserver/duplicati/) Docker containers. I make a distinction between the two because they use different mount points in their configurations.

That being said - during my 2.0.4.1 -> 2.0.3.14 downgrade test I found that after re-upgrading to to 2.0.4.1 I’m back to missing the 4 items listed previously (even though I did see them before) so this DOES SUPPORT your finding of seeing this oddity “sometimes”.

I tried 3 restarts of the official container and they have not yet returned.

I tested 2 restarts of the linuxserver container and they all showed up correctly every time.

When missing, they are also missing from the “Direct restore from backup files…” list.

Note that on my Windows 10 box (which still hows the v2 options in the web GUI) they still do not show up for me in the BackendTool help:

"C:\Program Files\Duplicati 2\Duplicati.CommandLine.BackendTool.exe" help
Usage: <command> <protocol>://<username>:<password>@<path> [filename]
Example: LIST ftp://user:pass@server/folder

Supported backends: aftp,amzcd,azure,b2,box,cloudfiles,dropbox,file,ftp,googledrive,gcs,hubic,jottacloud,mega,onedrive,openstack,s3,od4b,mssp,ssh,tahoe,webdav
Supported commands: GET PUT LIST DELETE CREATEFOLDER

@kenkendk or @Pectojin, can you think of any reason why some destination option would load sometimes but not others? I checked the DLLs in both official and linuxserver update and they seem to be the same.

kenkendk · November 12, 2018, 9:21am

I somehow missed this post completely. For the record, I upload images to https://hub.docker.com/r/duplicati/duplicati/, and I have no control over the linuxserver repo.

The onedrive and onedrivev2 backends are stored in the same dll Duplicati.Library.Backend.OneDrive.dll, so it is strange that one of the providers are missing.

I will investigate asap.

kenkendk · November 12, 2018, 10:10am

There seems to be some missing libraries in the Docker images:

> docker exec duplicati mono /opt/duplicati/Duplicati.CommandLine.exe test abc:// --no-encryption --console-log-level=explicitonly
Failed to load assembly /opt/duplicati/Duplicati.Library.OAuthHelper.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Microsoft.SharePoint.Client.Runtime.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/AWSSDK.Core.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Duplicati.Library.Backend.OneDrive.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Duplicati.Library.OAuthHelper.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/MegaApiClient.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Microsoft.SharePoint.Client.Runtime.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Microsoft.SharePoint.Client.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/AWSSDK.Core.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Duplicati.Library.Backend.OneDrive.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Duplicati.Library.OAuthHelper.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/MegaApiClient.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Microsoft.SharePoint.Client.Runtime.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.
Failed to load assembly /opt/duplicati/Microsoft.SharePoint.Client.dll, error message: Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown. => Exception of type 'System.Reflection.ReflectionTypeLoadException' was thrown.

kenkendk · November 12, 2018, 11:54am

I tracked down the problem. These two packages are missing from the Docker images:

libmono-system-net-http-webrequest4.0-cil
libmono-system-web4.0-cil

Edit:
You can “fix” the images by running these two commands (assuming your container is named duplicati):

> docker exec duplicati apt update
> docker exec duplicati apt install libmono-system-net-http-webrequest4.0-cil libmono-system-web4.0-cil -y

Then restart the container and all should work. I will update the docker scripts and build a new experimental.

kenkendk · November 12, 2018, 12:18pm

I have now put up the 2.0.4.2 experimental build, which is the same as 2.0.4.1, except with extra dependencies installed in the Docker containers.

ts678 · November 12, 2018, 1:51pm

I ran the fix (there’s a space-for-dash typo in “libmono-system-net-http webrequest4.0-cil”) and it worked fine. Also tried new container with duplicati/duplicati:linux-amd64-2.0.4.2_experimental_2018-11-12 however @JonMikelV in Release: 2.0.4.2 (experimental) 2018-11-12 raises a good question about upgrade.

My testing used the kenkendk CommandLine “test” test, my BackendTool “help” test (27 keys), and a browser, which shows 29 entries in the dropdown, because keys for openstack and s3 are also in “Standard protocols”.

I’m also not able to get the Windows 10 problem @JonMikelV reported above, at least up through 2.0.3.14 canary, running native on Windows 10. There’s a Docker for Windows, but that output snipping looks native…

JonMikelV · November 12, 2018, 1:54pm

Sorry for not clarifying that I’m running on W10 native - no Docker there.

kenkendk · November 12, 2018, 3:39pm

Ok, from the release post, I assumed you were running some Docker stuff.

But good question about the updates. The packages that I added were a part of the “host” operating system packages. The Duplicati updater does not touch the system in runs on.

Generally, when using Docker, I guess it makes sense to always update the container instead of using the built-in updater?

JonMikelV · November 12, 2018, 3:54pm

That makes sense but then wouldn’t we need to have 4 containers available - one for each of the release paths?

I’ll try installing another container now but my guess is I’ll end up with the 2.0.3.3 beta…

Pectojin · November 12, 2018, 4:05pm

Yup, most people will probably not even have the updates folder persisted, so a container restart would revert to the image version.

Ideally

JonMikelV · November 12, 2018, 4:15pm

Before I do something I’ll regret - do you know how the container update process works? I’ve set my unRAID box to check for Docker updates Daily with a time 5 min. in the future and it’s not finding an update for Duplicati.

I could manually re-install, but I was hoping the container updater would take care of things for me…

Pectojin · November 12, 2018, 4:18pm

I don’t run Duplicati in a container currently, but I’d have the container configured with it’s databases mounted to either a persistent volume or just straight out to the host disk. Then just destroy the container and deploy the new image to update. The new container will upgrade the databases on startup.

JonMikelV · November 12, 2018, 4:23pm

Agreed - that’s how is should work, but I’m trying to avoid it for two reasons.

I’m still not sure I’ve got my container configured correctly so may lose all my configs (but that’s OK - I’ve exported everything just in case)
I get linuxserver Duplicati container updates fairly frequently - I’ve NEVER gotten an official Duplicati container update.

I always assumed the lack of updates was because the container was the beta only, but even with 2.0.3.3 beta I didn’t get a container update. So I suspect there may be an issue with how we’re hosting or updating the container that isn’t advertising the fact. (Though again, that could just be me using containers incorrectly.)

Pectojin · November 12, 2018, 4:32pm

I vaguely recall LinuxServer stating somewhere they have a policy about canary releases. Maybe. My guess is they only update to the version on the website (beta 2.0.3.3) and every other release they do is just OS updates.

JonMikelV · November 12, 2018, 6:32pm

Wow - seeing my typos quoted by somebody else is almost as bad as hearing my own voice being played back.

I don’t disagree with your view of LinuxServer probably being only beta and most of the updates I see being OS related. However, that doesn’t explain why I don’t recall ever seeing an official container update.

Assuming the https://hub.docker.com/r/duplicati/duplicati/tags/ page reflects an “update” to the container, I should have seen 18 updates coming in. I just checked my update notification emails from my unRAID box and I’ve gotten 0 about the official Duplicati container.

Anyway - since no update was “detected” by unRAID I used it’s “Force update” feature to update the docker. As expected, it has reverted to version 2.0.3.3 beta of Duplicati - but I’m not sure which one as it won’t start due to a The database has version 6 but the largest supported version is 4. error. D’Oh!

So at this point I would recommend ~~@kenkendk’s manual fix over updating the docker container if you are currently running anything NEWER than 2.0.3.3 (which I think is the last one with database version 4)~~ understanding how to use Docker tags to make sure you’re using at least base container version 2.0.4.2.

Note that forcing the container update (resulting in 2.0.3.3. beta) then using the Duplicati updater to get to 2.0.4.2 experimental did NOT bring in the the missing Microsoft providers.