If one could specify the output format of a result it would be friendlier to those want do more with it in the “run-script-after”, the HTTP report module, and/or the Jabber. I don’t think it would be useful to change the formatting within the send email functionality.
It looks straightforward to implement and I would like to contribute such a change back to the repo.
As I start I’d add JSON.
What do other people think?
I think I should welcome you to the forum and say that it’s great that you’re offering to help improve Duplicati!
Would the JSON-ized output be an optional (parameter driven) thing or “always on” (in which case I wonder how many already existing scripts would break)?
Thanks!
I would think it’d be an optional parameter so that it is not a breaking change. As well, parameter seems more appropriate than a flag in the likely event that formats other than JSON are desired.
If the provided value is unsupported, either it defaults to the current format or fails. I’m leaning towards the former.
Agreed - current format as default, but it should also probably be a provided option. Are you thinking of a text field or a select list and do you know how to get it working in the GUI?
it should also probably be a provided option
Makes sense.
Are you thinking of a text field or a select list
UI-wise I’m thinking a list.
do you know how to get it working in the GUI
Not yet. I’ve only explored a few areas of the code, but it looks like this PR might be a good guide to get me started.
@JonMikelV I believe I have the functionality added.
What’s the project’s standard on unit testing?
Also, how do I build and test it? I want to be sure it works before opening a pull request.
There is some standard testing built into the pull request, though TBH I don’t know if it’s unit, regression, or misc. testing being done.
If you build (such as in Visual Studio) then the resulting files in the \bin\debug (or release) folder can be executed as is.
They will automatically run in portable more so all settings & such will stay in the same location folder (in other words it won’t mess with any normal install you might have running on the machine - as long as you don’t share destination folders).
I usually just run the tray icon and use that to open the browser UI, that way I don’t have to hunt around for (or specify) the port number.
Great! It has been on my “small things” wishlist since the report feature was introduced.
There is a basic functionality test in Travis, so each pull request is tested that it still works (backup, restore, repair, etc).
For a features such as this, it would be great if you could write a unittest for it, but since it does not impact core functionality I would not require it.
That sounds fair. If you use the Enumerable as the type for the option, the input is validated automatically and a warning is emitted if the value is not one of the listed options.
This is just what I needed. Thanks @JonMikelV! The UI displays my new command line option and displays the valid values in a dropdown list. I’ll test that both serializers work soon.
I’ll add one a few for sure. Would you be opposed to me adding a new project called “Duplicati.Library.Modules.Builtin.UnitTest”, containing only unit tests of the “Duplicati.Library.Modules.Builtin” project?
Does that imply that the user is not allowed to save an invalid option within the GUI?
What’s the behavior if a user uses an invalid option on the command line? Does it automatically pass RunScript, for example, the enumeration value that is set as the default for that command line option?
Not opposed, but what does it gain, as opposed to just dropping a new file like this:
I assume that at some point I (or someone else) will mess up the commandline options. Preventing the user from entering a wrong value would make a frustrating user experience. Instead, the UI is designed to not let you pick invalid choices, but it is always possible to enter invalid stuff in “text mode”.
This also allows for more smooth upgrades where an option changes, as it still allows you to save the backup.
All option values are passed simply as strings in a dictionary, so you need to parse the strings yourself in the module.
In retrospect it would have been better to allow the module to define a class with options, and then fill that before invoking the module as it would reduce the need for parsing code and enforce that all modules parse the same way.
But as it is now, the invalid string is passed to the module, but there is a checker in Controller.cs that emits a warning if the option value does not match the description.
A few things:
Separate question. While I was testing my changes, there was a condition that I did not handle. The Duplicati UI indicated that there were warnings, but nowhere in the UI could I find them. I was not until I set --log-file was I able to see the actual issue. Should I be seeing this in the UI as well? If so, is there a setting to change?
Also, here’s the first result from succeeding with JSON conversion. There are some things logged that are not currently logged (e.g. BackedStatistics, BackendWriter, MessageSink, etc.). I’m wondering if I should look at stripping those out?
{
"DeletedFiles": 0,
"DeletedFolders": 0,
"ModifiedFiles": 0,
"ExaminedFiles": 1,
"OpenedFiles": 0,
"AddedFiles": 0,
"SizeOfModifiedFiles": 0,
"SizeOfAddedFiles": 0,
"SizeOfExaminedFiles": 2261057,
"SizeOfOpenedFiles": 0,
"NotProcessedFiles": 0,
"AddedFolders": 0,
"TooLargeFiles": 0,
"FilesWithError": 0,
"ModifiedFolders": 0,
"ModifiedSymlinks": 0,
"AddedSymlinks": 0,
"DeletedSymlinks": 0,
"PartialBackup": false,
"Dryrun": false,
"MainOperation": "Backup",
"CompactResults": null,
"DeleteResults": null,
"RepairResults": null,
"TestResults": {
"MainOperation": "Test",
"Verifications": [
{
"Key": "duplicati-20180123T225148Z.dlist.zip",
"Value": []
},
{
"Key": "duplicati-ic59217ecf022474588ff70eaa219152f.dindex.zip",
"Value": []
},
{
"Key": "duplicati-b19f77f2d1f7d4181894a68ca40602c66.dblock.zip",
"Value": []
}
],
"VerboseOutput": false,
"VerboseErrors": false,
"ParsedResult": "Success",
"EndTime": "2018-01-24T00:02:15.4852709Z",
"BeginTime": "2018-01-24T00:02:15.1339592Z",
"Duration": "00:00:00.3513117",
"MessageSink": null,
"Messages": null,
"Warnings": null,
"Errors": null,
"BackendStatistics": {
"RemoteCalls": 5,
"BytesUploaded": 0,
"BytesDownloaded": 1480010,
"FilesUploaded": 0,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 3,
"KnownFileSize": 1480010,
"LastBackupDate": "2018-01-23T19:21:48-03:30",
"BackupListCount": 1,
"TotalQuotaSpace": 249690058752,
"FreeQuotaSpace": 139679543296,
"AssignedQuotaSpace": -1,
"MainOperation": "Backup",
"VerboseOutput": false,
"VerboseErrors": false,
"ParsedResult": "Success",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2018-01-24T00:02:13.8148129Z",
"Duration": "00:00:00",
"MessageSink": null,
"Messages": null,
"Warnings": null,
"Errors": null
},
"BackendWriter": {
"RemoteCalls": 5,
"BytesUploaded": 0,
"BytesDownloaded": 1480010,
"FilesUploaded": 0,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 3,
"KnownFileSize": 1480010,
"LastBackupDate": "2018-01-23T19:21:48-03:30",
"BackupListCount": 1,
"TotalQuotaSpace": 249690058752,
"FreeQuotaSpace": 139679543296,
"AssignedQuotaSpace": -1,
"MainOperation": "Backup",
"VerboseOutput": false,
"VerboseErrors": false,
"ParsedResult": "Success",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2018-01-24T00:02:13.8148129Z",
"Duration": "00:00:00",
"MessageSink": null,
"Messages": null,
"Warnings": null,
"Errors": null
}
},
"ParsedResult": "Success",
"VerboseOutput": false,
"VerboseErrors": false,
"EndTime": "2018-01-24T00:02:15.8269689Z",
"BeginTime": "2018-01-24T00:02:13.8138123Z",
"Duration": "00:00:02.0131566",
"MessageSink": {
"QuietConsole": false,
"VerboseOutput": false,
"VerboseErrors": false,
"Output": {
"Encoding": {
"m_isReadOnly": true,
"encoderFallback": {
"MaxCharCount": 1
},
"decoderFallback": {
"MaxCharCount": 1
},
"m_codePage": 850,
"dataItem": null,
"Encoding+m_codePage": 850,
"Encoding+dataItem": null,
"maxCharSize": 1
},
"FormatProvider": "en-US",
"NewLine": "\r\n"
},
"BackendProgress": {},
"OperationProgress": {}
},
"Messages": [],
"Warnings": [],
"Errors": [],
"BackendStatistics": {
"RemoteCalls": 5,
"BytesUploaded": 0,
"BytesDownloaded": 1480010,
"FilesUploaded": 0,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 3,
"KnownFileSize": 1480010,
"LastBackupDate": "2018-01-23T19:21:48-03:30",
"BackupListCount": 1,
"TotalQuotaSpace": 249690058752,
"FreeQuotaSpace": 139679543296,
"AssignedQuotaSpace": -1,
"MainOperation": "Backup",
"VerboseOutput": false,
"VerboseErrors": false,
"ParsedResult": "Success",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2018-01-24T00:02:13.8148129Z",
"Duration": "00:00:00",
"MessageSink": null,
"Messages": null,
"Warnings": null,
"Errors": null
},
"BackendWriter": {
"RemoteCalls": 5,
"BytesUploaded": 0,
"BytesDownloaded": 1480010,
"FilesUploaded": 0,
"FilesDownloaded": 3,
"FilesDeleted": 0,
"FoldersCreated": 0,
"RetryAttempts": 0,
"UnknownFileSize": 0,
"UnknownFileCount": 0,
"KnownFileCount": 3,
"KnownFileSize": 1480010,
"LastBackupDate": "2018-01-23T19:21:48-03:30",
"BackupListCount": 1,
"TotalQuotaSpace": 249690058752,
"FreeQuotaSpace": 139679543296,
"AssignedQuotaSpace": -1,
"MainOperation": "Backup",
"VerboseOutput": false,
"VerboseErrors": false,
"ParsedResult": "Success",
"EndTime": "0001-01-01T00:00:00",
"BeginTime": "2018-01-24T00:02:13.8148129Z",
"Duration": "00:00:00",
"MessageSink": null,
"Messages": null,
"Warnings": null,
"Errors": null
}
}
I wish it would be standardized, because current HTTP post value isn’t.
I’ll check if there’s some method to convert it to JSON on the server side (PHP probably).
I have only one question: when?)) When it will in D2?))
Any progress ? I would like to send the results in an a database to graph the data with grafana.
It’s pretty much completed, as per my previous message. I was side tracked for a bit but now I can refocus on it. You can look at the following branch if you want to play around with it yourself: GitHub - StephenGregory/duplicati at export-result-into-different-formats.
Hoping to test that my changes are compatible with the latest release, do some clean up, and create a PR within the next week.
I just added a bunch of fixes to the reporting code as well, such that it supports the new log system:
It clashes a bit with your changes, but I like the idea that we can have individual serializers. I did a quick hack to support sending JSON from the HTTP module, but I think we can use your serializer work in my update.
Would you like to merge in your changes first, and let me deal with the conflicts?
I just added a bunch of fixes to the reporting code as well, such that it supports the new log system
Great!
Would you like to merge in your changes first, and let me deal with the conflicts?
I wouldn’t mind getting these in first.
The final thing I am looking at is excluding some properties that I think are irrelevant from being serialized. What implementations of Nevermind, figured something out.IBasicResults are serialized in RunScript, SendHttpMessage, etc? Is there an easy way to determine that?