Release: 2.0.4.4 (canary) 2018-11-14

kenkendk · November 14, 2018, 7:48am

2.0.4.4-2.0.4.4_canary_2018-11-14

Added password confirmation on change, thanks @LacunaSoftware
Fixed a crash with VSS, thanks @verhoek

Taomyn · November 14, 2018, 8:13am

Updated and the VSS error has now gone, backup was clean Will update a couple of other machines later

So yes, many thanks @verhoek for that fix

Now to see how enabling USN goes, hoping that the long path name errors have gone…

ts678 · November 14, 2018, 2:16pm

Thanks @verhoek. The original problem seems fine, and now I’ve stumbled over another odd one where the same job works fine in web UI “Run now” but fails from the web UI “Commandline” option, without any changes:

Failed: Value cannot be null.
Parameter name: source
Details: System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector)
   at Duplicati.Library.Utility.FilterExpression.GetFilterHash()
   at Duplicati.Library.Snapshots.UsnJournalService.Initialize(IFilter emitFilter, IEnumerable`1 prevJournalData)
   at Duplicati.Library.Main.Operation.BackupHandler.GetJournalService(IEnumerable`1 sources, ISnapshotService snapshot, IFilter filter, Int64 lastfilesetid)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)

Log data:
2018-11-14 09:08:15 -05 - [Error-Duplicati.Library.Main.Operation.BackupHandler-FatalError]: Fatal error
System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector)
   at Duplicati.Library.Utility.FilterExpression.GetFilterHash()
   at Duplicati.Library.Snapshots.UsnJournalService.Initialize(IFilter emitFilter, IEnumerable`1 prevJournalData)
   at Duplicati.Library.Main.Operation.BackupHandler.GetJournalService(IEnumerable`1 sources, ISnapshotService snapshot, IFilter filter, Int64 lastfilesetid)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()

Originally I was wondering if it involved --snapshot-policy, so I took that out. Above has --usn-policy=Required.

verhoek · November 14, 2018, 3:07pm

Did this also happen in the previous version, say two canaries before?

Can you make another github issue? I see the issue from a technical point of view - that anyway needs to be fixed. I will fix that and see what Kenneth says about it from a functional point of view.

ts678 · November 14, 2018, 3:20pm

Seems like yes, although I haven’t tried --usn-policy in awhile. I moved 2.0.4.4 and 2.0.4.3 aside between the two runs below (adjusted to be more minimal). There’s also a small chance my config caused this. More later.

C:\>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" backup "file://C:\Duplicati Backups\local test 6\\" "C:\backup this\short.txt" --dbpath="C:\ProgramData\Duplicati\80858365726684876975.sqlite" --encryption-module=  --no-encryption=true --disable-module=console-password-input --usn-policy=Required
Backup started at 11/14/2018 10:10:26 AM
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  1 files need to be examined (8 bytes)
Fatal error => Value cannot be null.
Parameter name: source

System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector)
   at Duplicati.Library.Utility.FilterExpression.GetFilterHash()
   at Duplicati.Library.Snapshots.UsnJournalService.Initialize(IFilter emitFilter, IEnumerable`1 prevJournalData)
   at Duplicati.Library.Main.Operation.BackupHandler.GetJournalService(IEnumerable`1 sources, ISnapshotService snapshot, IFilter filter, Int64 lastfilesetid)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.CommandLine.Commands.Backup(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

C:\>"C:\Program Files\Duplicati 2\Duplicati.CommandLine.exe" backup "file://C:\Duplicati Backups\local test 6\\" "C:\backup this\short.txt" --dbpath="C:\ProgramData\Duplicati\80858365726684876975.sqlite" --encryption-module=  --no-encryption=true --disable-module=console-password-input --usn-policy=Required
Backup started at 11/14/2018 10:15:09 AM
Checking remote backup ...
  Listing remote folder ...
Scanning local files ...
  1 files need to be examined (8 bytes)
Fatal error => Value cannot be null.
Parameter name: source

System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector)
   at Duplicati.Library.Utility.FilterExpression.GetFilterHash()
   at Duplicati.Library.Snapshots.UsnJournalService.Initialize(IFilter emitFilter, IEnumerable`1 prevJournalData)
   at Duplicati.Library.Main.Operation.BackupHandler.GetJournalService(IEnumerable`1 sources, ISnapshotService snapshot, IFilter filter, Int64 lastfilesetid)
   at Duplicati.Library.Main.Operation.BackupHandler.<RunAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at CoCoL.ChannelExtensions.WaitForTaskOrThrow(Task task)
   at Duplicati.Library.Main.Controller.<>c__DisplayClass13_0.<Backup>b__0(BackupResults result)
   at Duplicati.Library.Main.Controller.RunAction[T](T result, String[]& paths, IFilter& filter, Action`1 method)
   at Duplicati.Library.Main.Controller.Backup(String[] inputsources, IFilter filter)
   at Duplicati.CommandLine.Commands.Backup(TextWriter outwriter, Action`1 setup, List`1 args, Dictionary`2 options, IFilter filter)
   at Duplicati.CommandLine.Program.ParseCommandLine(TextWriter outwriter, Action`1 setup, Boolean& verboseErrors, String[] args)
   at Duplicati.CommandLine.Program.RunCommandLine(TextWriter outwriter, TextWriter errwriter, Action`1 setup, String[] args)

C:\>

verhoek · November 14, 2018, 3:22pm

It looks like it was already there and could be due to an incorrect filter setting of some sort, but still, it may not crash like that.

ts678 · November 14, 2018, 6:45pm

Running the same command going backwards in versions, the bug seems begin in 2.0.3.8. 2.0.3.7 was OK, so maybe I should be asking @dgehri (I wonder if the forum knows about the GitHub names that aren’t in forum)?

kenkendk · November 14, 2018, 9:05pm

It does not. Some users just happen to have the same name on the forum and on Github.

henryg · November 16, 2018, 2:50pm

Gave this a try and it would not work with my existing OneDrive (Office 365) account/backups. Tried re-auth’ing but no joy.

Thankfully when I reverted to 2.0.4.2_experimental_2018-11-12 all was ok. Phew :-}

JonMikelV · November 16, 2018, 3:18pm

I wonder if OneDrive (Office 365) uses the v1 API that Microsoft (say they) retired so has been removed from Dupliacti starting with Release: 2.0.4.3 (canary) 2018-11-13?

@kenkendk, I compared the System Info “backendgroups” and “Backend Modules” between 2.0.4.2 and 2.0.4.4 and it looks like the non-v2 “onedrive” is removed from the “Backend modules” but is still listed in “backendgroups”. Is that supposed to still be there?

Here’s the “backendgroups” for 2.0.4.4 (which is the same for 2.0.4.2):

backendgroups: {
	"std": {
		"ftp": null,
		"ssh": null,
		"webdav": null,
		"openstack": "OpenStack Object Storage / Swift",
		"s3": "S3 Compatible",
		"aftp": "FTP (Alternative)"
	},
	"local": {
		"file": null
	},
	"prop": {
		"s3": null,
		"azure": null,
		"googledrive": null,
		"onedrive": null,   <-----------------
		"onedrivev2": null,
		"sharepoint": null,
		"msgroup": null,
		"cloudfiles": null,
		"gcs": null,
		"openstack": null,
		"hubic": null,
		"amzcd": null,
		"b2": null,
		"mega": null,
		"box": null,
		"od4b": null,
		"mssp": null,
		"dropbox": null,
		"sia": null,
		"jottacloud": null,
		"rclone": null
	}
}

henryg · November 16, 2018, 5:38pm

Does this mean we 365 users will lose the ability to backup to OneDrive? I certainly can’t update further until this is resolved.

OneDrive backup is a virtue that most programs I looked at pre-Duplicati lacked.

kenkendk · November 20, 2018, 9:18am

That is possible. The documentation for the Office 365 API says to use the Graph API, which is what OneDrive v2 is using:

It should be removed. I think there is still some OneDrive v1 code in the js and html parts. It should not be visible to the user, but we should remove it anyway.

Are you using the onedrive://... destination for the Office 365 connection? I hope that onedrivev2://... works for 365 users?

henryg · November 20, 2018, 11:35am

When I switched the destination via the GUI to Storage Type OneDrive v2 and tested the connection, it failed to connect (details below). I exported the job prior to the change and the json file shows I was/am using onedrive://…for v1

Would the AuthID need changing if moving from v1 to v2?

I will stick with the “Experimental” release until the issue can be resolved.

v2 test connect error report:
Failed to connect: Unauthorized: Unauthorized error from request https://graph.microsoft.com/v1.0/me/drive/root:/Duplicati Method: GET, RequestUri: ‘https://graph.microsoft.com/v1.0/me/drive/root:/Duplicati’, Version: 1.1, Content: <null>, Headers: { User-Agent: Duplicati/2.0.4.2 Authorization: Bearer ABC…XYZ } StatusCode: 401, ReasonPhrase: ‘Unauthorized’, Version: 1.1, Content: System.Net.Http.StreamContent, Headers: { request-id: 362025cf-b4d8-48a5-94ad-6034e38fc19b client-request-id: 362025cf-b4d8-48a5-94ad-6034e38fc19b x-ms-ags-diagnostic: {“ServerInfo”:{“DataCenter”:“UK South”,“Slice”:“SliceC”,“Ring”:“5”,“ScaleUnit”:“002”,“Host”:“AGSFE_IN_8”,“ADSiteName”:“UKS”}} Strict-Transport-Security: max-age=31536000 Date: Tue, 20 Nov 2018 11:22:54 GMT WWW-Authenticate: Bearer realm="", authorization_uri=“Sign in to your account”, client_id=“00000003-0000-0000-c000-000000000000” Content-Length: 265 Content-Type: application/json; charset=utf-8 } { “error”: { “code”: “InvalidAuthenticationToken”, “message”: “CompactToken parsing failed with error code: 8004920A”, “innerError”: { “request-id”: “362025cf-b4d8-48a5-94ad-6034e38fc19b”, “date”: “2018-11-20T11:22:54” } } }

kenkendk · November 20, 2018, 11:45am

Yes! The Graph API is using its own OAuth service with MS, so you need to re-auth to use the Graph API.

I was worried that we would have to postpone the upgrade from experimental to beta, but re-reading the thread I can see that it is only the removal of OneDrive v1 that is the problem.

If v1 still works for 365, I don’t think we should remove it yet. I would like to rename it though, such that new (non-365) users do not start out by trying v1 and conclude that it does not work :).

henryg · November 20, 2018, 12:41pm

Sorry, it was my ignorance that caused the problem I set up a (small) OneDrive v2 test job, got a new AuthID, and it worked fine inc restore.

If I change my current job from v1 to v2 using the new AuthID will there be any consequences, eg it would try to backup the whole lot from scratch, or loss or file versions. I’m guessing not but…

ts678 · November 20, 2018, 1:32pm

Do we know this stopped working for anyone? Yesterday’s stats at https://usage-reporter.duplicati.com/ show 2947 OneDrive backups, and I wonder if the Graph API statistics could be shown. I assume they’re collected, however maybe they’re down there in the small Other count. I don’t think Duplicati version info is listed there. Basically I’m wondering how much exposure the v2 stuff is getting even after having it in 2.0.4.2 experimental.

And I just set up a (small) OneDrive test job after going to 2.0.4.2 and it worked for me, at least for right now.

Microsoft’s similar product names often confuse me. I previously went through Office 365 Group setup far enough to see it use msgroup:// for its URL protocol prefix (visible in job Commandline option), but looking further, I think SharePoint is its storage part, so this is like OneDrive versus OneDrive for Business split.

The commandline help for msgroup seems to say it needs sharepoint prefix (isn’t that self-contradictory?), and suggests using --site-id, which the UI for Office 365 Group doesn’t actually list as being available.

Someone who actually knows the Microsoft plan should probably review the help and Storage Providers info, which sound like they may need changes anyway in an effort to keep people from beginning on retiring APIs. Helpful migration hints would also be a good thing to put in the manual (or at least in a How-To for the forum).

henryg · November 20, 2018, 7:10pm

Changed ‘proper’ job to v2 and all seems ok - verified ok and took expected time to backup, and test restore worked

torsten · November 20, 2018, 8:52pm

OK, changing the destination to OneDrive v2 worked for me.

But it didn’t work by simply changing the destination. I had to remove the old authid in the pro options right below the button to check the connection. Without deleting that option, I only received an error message.