Client-side agent and centralized management / dashboard

Features
1

Hello,

First, I’d like to say, I like your product a lot! You’ve done GREAT job with client-side backup system which supports EVERYTHING a user needs. What I mostly like, is that you can use hell a lot different destinations, I personally use Amazon S3, but also OneDrive, several NAS, FTP etc.

What this product is lacking - centralized management. I’m thinking of creating one.

Since I’m not a pro-coder, I will not know all the specifics and nowadays “correct” methods of dealing something, but I am capable of programming and generating solutions.

First, centralized management needs client-side agent which communicates with central server (web-server probably, easiest to host and also probably to manage).
I have not decided which program language I should use for client-side agent. I do not want to make that mistake that it’s not possible to release client to Linux as well (I am a Windows man) and therefore I’d like your suggestion. Which programming language should I use in order to program language on Windows, but in future for Linux and other OSes as well? Something universal probably, not sure yet.

For client I thought:

  • Client does all the “on demand” communications, which means client communicates with central server, not other way. Why? Because of port-forwarding. I do not want to forward any ports and therefore client opens all the connections and server just answers.
  • Since I’ve never done it before, I do not know communication “standards”, but I’m capable of learning. I thought of JSON API on the server side and client-agent asks all the commands from server.
  • Commands are straight forward - client-agent just asks for commands and delivers them to Duplicati REST API. Client-agent does not consist of any commands other than authentication (I guess every API needs somekind of authentication).

So, basically, process is like that:

  1. Client1 client-agent asks server for commands;
  2. Server answers to client-agent request, for example sends “update last backup time”;
  3. Client-agent gets the command, delivers it to Duplicati REST API;
  4. Client-agent gets an answer from Duplicati REST API;
  5. Client-agent delivers the answer to server;
  6. Server now has updated “last backup time” for client1;
    Everything repeats

Now, on the server side, I thought about simple PHP+MySQL. Since I’m not a designer, this product will probably look ugly.
At first, I’m probably going to make simple monitoring system (and reporting if anything goes wrong or doesn’t work) and later on, I’ll add client configuration options, group editing etc.

What do you think about it? Anyone already doing what I’m thinking of?

As I stated before - please help me to choose “correct” programming language for client-agent, for future multi OS.

Thank you!

1 Like
2

Hello @LKits, and welcome to the forum - it’s nice to hear you like Duplicati!

There are a few centralized reporting (not management) projects people have created including dupReport and Duplicati Monitoring, however neither provides “management”.

Deciding what sorts of features you are wanting to implement and frequency of communication between client and server can effect the technology choices you make. Are you looking at just centralized reporting / dashboard (as your example shows) or remote management where your server can do things like create / edit / start / stop / delete backup jobs?

3

Thank you for those suggestions - I did not know Duplicati has those advanced features and HTTP report URL is exactly what is needed for monitoring report.

“Duplicati Monitoring” is something I’m going to create in the beginning, but I’m trying to advance it further so one is able to change client Duplicati settings through dashboard as well.
I see they are offering nice (free) service and it’s all great, but not suited for me, because I am not able to install it on my own web server. I’m not saying I don’t trust them, but I’m saying I’ve seen too many discontinued services in the past that downloadable and installable products are better than services.

Anyways, I’ll check out Duplicati Monitoring capabilities and probably will start Github project or something.

Once again, thanks very much!

1 Like
4

You’re welcome - good luck!

5

A HTTP based monitoring service like https://duplicati-monitoring.com that you can host yourself would be more than welcome.

Duplicati-Monitoring is very easy to setup and will get you started quickly. Users that want to keep logfiles on their own hardware and/or whitelabel the software, could benefit from a self hosted solution.

6

If you’re looking for self-hosted solution, you might want to check out dupReport. It’s an email collector/analyzer rather than a web site reporter, but it is self-hosted and light-weight, so it might fit your needs rather than building something yourself.

1 Like
7

Thanks for pointing that out. I forgot to mention your solution, which indeed could fit the OP’s needs. But a self-hosted http based Duplicati-Monitoring alternative is still welcome.

8

Agreed. BTW, just curious, do you run duplicati-monitoring.com from your own servers or is it hosted somewhere?

9

I’m not involved in Duplicat-Monitoring development in any way, just a happy end user!
@Tekki is the one to ask this question to.

10

I think it’s only available as hosted. I couldn’t find any download links or source code. I really wanted to spin it up locally :confused:

11

Nope, it wasn’t me! ^^

12

Excuse me, my bad!
@crazy4chrissi developed and maintains Duplicati Monitoring.

13

Hey guys,

Duplicati-Monitoring is a service developed, provided and hosted by Mittelstandsoptimierer. Vertumno GmbH. We are a German company providing different services including hosting, web-development and cloud backup. We respect your data. As a German company, we have to (and want to) meet German (and European) data privacy laws, which are rather strict. Moreover, we are experts regarding data privacy, as we also offer data privacy audits.
We are definitely not going to sell your data. We don’t even like the idea to finance the service through ads. This is why we chose to finance the service through donations. Please donate so we can keep the service free of ads.

The service is not hosted on some cloud space. The server that we are using to host the service is a dedicated server managed by ourselves. It is located in a German data center operated by a German company that we have carefully selected and that is also ISO/IEC 27001 certified.

We carefully treat your data, even though the data that our service collects usually does not contain big secrets. The most confidential data is contained in the error messages that Duplicati produces. It may contain paths and file names that might be confidential. If you see anything confidential in the reports, you can delete the report.

No, there is no download option, yet. We may offer licenses of the software so you can host it yourself in your company - contact me directly if anyone is interested. We love open source and we might make the software open source at some point. In the development, we carefully made sure that no components are used which could not be released as open source, so we keep this option open. But first, it has to pay off. Sorry, we are a company, and we need to pay our bills and employees.

If you need a self-hosted open source solution, I’d recommend dupReport by @handyman .

Greetings,
Christopher

2 Likes
14

One more thing: I know lots of you fear that we might just stop our service, just like crashplan did. But there are several reasons why we won’t:

  • we (our company) depend on it ourselves
  • I (lead developer and head of programming department) use the service myself and have a private interest in it
  • the service is very successful, new users register each day and the number of backup reports is continuously increasing
  • if money would be a problem (not enough donations), we would rather turn it into freemium or ad-based service than stopping it. In that case, there will always be an option for private users to use it for free, maybe with adverts then.
  • in the unlikely event that our company lost interest in the whole project, we would rather make our git repo public and let the open source community take over, than dump our work into the wastebin.

Don’t worry, we are not like Crashplan. They offered the service for free and had high costs per user for the storage. We offer the service for free, but finance it through donations, and don’t have such high costs per user. They could not release the computer-to-computer backup functionality as it contains proprietary technology. We only used components that can be released as open source right from the start.

We will soon publish some more information on how much donations we expect to keep the service running and a status page that says for how long in the future the donations ensure that the service stays free and ad-free.

We believe in the donation-model as we know some of our users can easily afford to donate some money, whereas others just can’t. With the donation model, everybody can donate as much as he can afford and what the service is worth for him. From what we have seen so far, this model can work out. If it doesn’t, we still have the option to introduce ads or make it a freemium service.

1 Like
15

Thank you for your answers.

I respect your business model. Still, I believe, when a product like Duplicati is free for customers, all the addons should either be free or not, still downloadable and off-site products, non-dependent on the developer hosted service. I understand your promises, let’s just say, my own experience instructs me to act differently.

I will probably find open-source HTML5 dashboard template and will build monitoring system on it. It’s not that hard, needs structural build-up, few mysql database tables.

16

I really like the (still free) service that @crazy4chrissi offers. It’s easy to set up, configured in a few minutes and gives a nice global overview of the status of all your configured backup jobs. Hosting is part of the service, there are no prerequisites for the end user.

However, anyone who wants to whitelabel Duplicati for offering an online backup solution commercially, probably wants to host a monitoring service themselves. @handyguy’s dupReport could be a nice tool in this scenario, but if you want a solution based on HTTP reporting, or if you want more flexibility in (the number of) reports you receive, an open source solution would be more than welcome. So @LKits, keep us updated!

17

@LKits: Of course, if you prefer a self-hosted solution, go ahead and build your own monitoring service. Just some words of advice: Yeah, I thought as well it is something not that hard to code. But it turned out that it is a lot more complex than I had thought in the beginning. It was really fast until we got a “working” solution (I have built a couple of systems like that), but since then, a lot of problems came up that I did not expect.

I have heard the same thing from users of our service that had started to code their own monitoring solution and finally stopped because maintaining it was too much effort. This is a quote from one user:

I don’t know how many hours you have spent on this project, […] I’ve spent waaaayyyyy too long. I appreciate what you’ve done[…]

@handyguy how much time do you spend on dupReport? Did you expect that much work?

Don’t get me wrong, I don’t want to stop you coding your own solution. Please code your own solution and make it open source. But be prepared that you (or someone else) will need a lot of time to maintain the solution, fix bugs, adjust to new Duplicati versions, etc. This is nothing that you code within one day and then its done. And it helps nobody if you are the 10th guy who starts a project like this and stops early.

@kees-z: Anybody who is interested in a branded duplicati-monitoring service, just contact me. We can offer you a branded version with your logo, your domain name, your product name. No problem.

Regarding more flexibility regarding number of reports: This is on our todo.

18

Great observations, @crazy4chrissi. dupReport started as a couple of shell & awk scripts I threw together in an afternoon to fit a need I had. Then I saw other Duplicati users had the same need and, well you know the rest. I spend far more time than I ever thought I would (or should) maintaining and updating the program. I love every minute of it, but occasionally I have to remind myself that I have other things to do, like a day job, a family, eating, etc.

1 Like
19

All solutions have their own pros and cons.

Duplicati-Monitoring pros: quick and easy: create an account and you’re ready to go, very nice visualization of backup results, daily email notifications, no own hardware needed that’s up 24/7 to receive HTTP messages…

dupReport pros: Source code available (can be tweaked by end user to fit personal needs), nice global overview of all backup results, can be hosted on almost anything, lightweight.

dupReport seems to be great for personal use. If you run backups for others (family, customers), I guess dupReport cannot split reports for different customers, unless you run multiple instances and use multiple email addresses to send reports to (not sure about this, is this correct?).
With Duplicati-Monitoring you can create an account for every customer. The possibility to whitelabel Duplicati-Monitoring is great, but I imagine that some people prefer to run it on their own hardware.

If @LKits can develop something like Duplicati-Monitoring that you can run on own hardware, it fills a gap between dupReport (great for personal use, self-hosted) and Duplicati-Monitoring (great for people who want to monitor backup jobs for themselves and/or customers, SaaS).

I don’t think a self-hosted tool like Duplicati-Monitoring and Duplicati-Monitoring itself suffer from each other: they serve different target groups. A self-hosted solution will be much more of a hassle than Duplicati-Monitoring: probably you have to setup a web server, SQL database etc. Using Duplicati-Monitoring, the hard part is already done for you: you only have to create an account and can start monitoring your backup jobs
.
Personally I think I’ll keep using Duplicati-Monitoring, but if I ever would offer it commercially, I guess I would prefer to set up my own server and host it myself.

20

Work in progress, reporting side is working great, now I need to add reporting (cron jobs, e-mails etc).
Currently I solved everything via tables, easy to sort. Probably will re-do it in the future so it’s easier on eyes, different clients in different categories etc.

l96s1qb.png1600×769 10.7 KB

2 Likes