Thanks for all the info @crazy4chrissi, the website left a bit to be desired in terms of information, which is why I was extra sceptical. I don’t know whether open sourcing the software will have any negative impacts on donations, but that’s of course up to you as the creators 
@LKits
Thank you for your work! Looks great already.
What do you think about (later on) start the duplicati jobs remotely on the clients?
Maybe
PSExec (Windows -> Windows)
WinExe (Linux -> Windows)
…
can be used to start duplicati on the client.
For security reasons I would like to have the credentials for accessing the backup backend transfered during this remote start. (I’m concerned, that in future malware might attact the duplicati client and extracts the connection parameters: see post #5 and following in (Security) feature request: Import and export config files “remotely” on a NAS).
At this moment, it’s monitoring only. I will be adding configuration part to it later. Monitoring is done through Duplicati itself (HTTP report url), but configuration must be done through installed client-agent.
I still haven’t decided which programming language I should use in order to release client on multiple OSes.
If one gets client configuration, then all they can do is see the report http url which is unique for absolutely every client. SQL injection theoretically is possible, in reality, using PDO prepare and execute (Google search those) somewhat eliminates that possibility. And through report url one does not have access to dashboard. Dashboard and all assets are protected by .htaccess password (at the moment, will decide if I want to make account management in future).
2 Likes
Looking great!
What platforms will it run on? If Linux-only, could is possibly run on a Raspberry Pi?
Any web server - at the moment it has HTML, javascript on client side and back-end PHP+MySQL.
And yes, RPi can run Apache.
I will build it a bit around, going to make everything through POST JSON API - makes things easier in the future.
1 Like
I’m curious. What’s the reasoning behind using a client-agent over simply using the REST API on each client machine?
Any action possible over the CLI should be possible over the API and if it isn’t right now it will eventually be. And any action possible through the Duplicati Web GUI is going through the API.
I looked at the API used by the client and the authentication scheme looks tricky to replicate. Has anyone written a client that queries the duplicati server directly using proper authentication? I’d love to see examples.
I think you’re out of luck. I haven’t seen any implementations.
There isn’t a restful way to authenticate right now, you basically get the index page and extract the token. If the page is password protected the client needs to catch the 302 redirect to /login.html and then send a POST request at /login.cgi with the hash value of the password.
I can write up an example for you in Python sometime this week.
Also I should really get around to implementing that restful auth method…
You might as well build an SSH server (or harden the current web server & add everything) into the Duplicati server as well. With that and easier to work with API remote management becomes much easier to implement…
Considering all the clever reporting tools people are building now, imagine what we could see with simple(ish)+secure remote access to the Duplicati server.
I think it depends on the use case.
If you want to build a central management dashboard then REST will give a lot of advantages. One being that it already exists, the other being that you know for sure that the Web GUI does exactly the same thing because it calls the same API.
That, and I’m basically a REST evangelist. If it doesn’t have a REST API I usually find something else 
I’m replacing all my shell scripts at work with Ansible so that I can get it into AWX/Ansible Tower and start doing SSH work through REST calls 
From a security point of view, using SSH has distinct disavantage. First you would need to create a user that can SSH into the machine. Then you would need to sudo or jail the access, so the user can’t do anything else. You’d still be opening yourself to all sorts of nasty attacks. Lets not even try to do any SSH automation on Windows …
The less people can SSH into a machine, the happier I am.
+1 for REST
1 Like
Think about it - how do you open a connection from remote location to client when they are not withing the same local subnet?
You’d need port forwarding.
So, instead of server connecting to client, the client needs to connect to server instead. That way one does not have to forward any ports on the client side, because server doesn’t open a connection to client.
And yes, REST API probably would be used, but right now I’m not familiar with it’s capabilities. REST API is just interface/system, what can you do with it, is another story.
Ah, yes that’s a good use case. Any computers moving around between networks would benefit from an agent. The agent could still be using the rest API locally but it would need to be there to initialize an outgoing connection.
I feel like I may be a bit slow on the uptake here, so please bear with me. Personally, I’m old school and feel comfortable with SSH, but if there’s newer more appropriate technology, that works for me. 
So should such an agent be part of Duplicati or part of whatever centralized reporting / management tool somebody decides to implement (like a proxy between the mgmt server and the client)?
I’d feel better if it’s baked in to Duplicati (assuming it’s not too big) so I’m not having to trust and install something else from somebody else on my clients. On the server? Sure - I can control that, but I’m less keen on doing that with all my clients.
So assuming it’s baked in, conceptually would this be additional parameters for the server (service / daemon or tray-icon) to do things like:
- configure the rprt/mgmt service destination (IP or URL…with forced SSL?)
- some hash or other key so rprt/mgmt server can be confident an authorized client is connecting
- frequency of attempted check-ins with the centralized rprt/mgmt service
- functionality allowed to be centrally shared/controlled (maybe some clients only want centralized reporting but not management)
Or am I completely off base of how this sort of thing would work?
But what you’d like is the ability to make CLI calls against the Duplicati server, right? Kind of in the direction #2693 is going?
I think it depends on exactly what we want to achieve whether or not it makes sense to bake it into the Duplicati server. I see multiple options to have a client control functionality on a machine.
- baked into server
- a separate process that runs either as a service or is periodically started by cron
- a service on a different machine that can control the Duplicati server on the target machine
When baked into the server, it would probably be a global parameter set on the server to have it periodically check a custom url for instructions and execute them. This functionality combined with http requests sent on job completion would give most management/monitoring dashboards full functionality.
I think the 2 other options, and the request for a CLI for the server, could theoretically share 50-60% of the same code if they rely on a REST API since much of it is parsing and sending data to the Duplicati API.
That kind of code reusability makes for a pretty strong case, in my book, to implement it all in one external tool. Easier to maintain than 3 separate tools and it requires no changes to the Duplicati core logic because it uses the Web GUI’s API, which makes it 4 different tools supported by the same interface.
yes, a master url, a preshared key, check-in frequency, and “allowed remote commands”.
Ideally, a client, regardless of which implementation, would just need those 4 parameters and then it has a defined format for a “to-do” list. The to-do list would then be pulled by the client and the tasks are then executed by the client.
That would allow any external dashboard/management tool to use the clients by just adhering to the to-do format.
I can get behind that if it really is ONE external tool, but I’m worried we could end up with a dozen different tools each doing a different thing and each “requiring” their own installation.
Plus there’s the “CrashPlan Effect” to worry about - to me double the number of code sources means double the opportunity for somebody to decide to retire and suddenly my slick “does everything I want” set of tools stops working (or at least being maintained).
I definitely would prefer for the client to be a Duplicati owned project and being the one option. Then letting anyone implement services that use the client’s interface.
I’m not saying it’s certainly the best option, but it seems like a good idea to consolidate those tools if possible.
As I mentioned on #2693, I don’t think it’s realistic to turn the existing CLI into a tool for managing the Duplicati server. They share most of their code, but the way they’re run they are completely separate and it would likely take a lot to make it optionally control the server.
I think it’s much more realistic to write a small command line tool that uses the same API as the GUI because that cuts out any need to maintain the interface. And it should be possible to turn that into a combined CLI, daemon, and cron-enabled tool supporting each of the use cases.
1 Like
Defining a check-in frequency will result in long wait times (setting it to once an hour) or unnecessary data traffic (setting it to every 5 seconds).
Is it technically possible to implement something similar to Microsoft Direct Push? This will instantly deliver commands to the server (a backup now-button will work without any delay), but will not generate any traffic when nothing changes.
I don’t know the exact specification, but it’s likely frequent small packages that ask if anything changed and then some logic that pulls the list when it changes. This should be easy enough to implement and if we’re only sending/requesting a few bytes it should be unnoticeable even at 1 request per second.
AFAIK, there is (almost) no traffic as long as nothing happens. This is accomplished by not closing the http session that the client initiates. Client connects to server and asks if something has changed. If nothing changed, the server doesn’t respond back, until there is a change.
When something needs to be updated, the server uses the open session to send a command to the client. From time to time, a heartbeat signal is sent to keep the session open.
This will result in applying updates instantly (clicking “backup now” would be immediately passed over to the Duplicati server), but a very small amount of data traffic (say once an hour instead of every second).
I couldn’t find very much in-depth information about Direct Push, this explains it a bit:
Mobile Messaging with Exchange ActiveSync