Is there a way for Duplicati to send me the progress data of the active backups as well as the results of the latest backups to my local server so that I can monitor them with my software. Perhaps I could during and at the end of the copy process run a remote script on my localhost to do this?
Thank you!
Ramiro
---------------------- Spanish
Existe alguna forma de que Duplicati me envie a mi servidor local los datos de progreso de las copias de seguridad activas así como el resultado de las últimas copias de seguridad para poder monitorizarlas con mi software. ¿Quizás pudiera durante y al finalizar el proceso de copia ejecutar un script remoto en mi localhost para hacer esto?
Gracias!
Ramiro
Duplicati-Monitoring.com is probably the easiest way to keep track of you backup statuses.
Otherwise you’d have to develop a tool that reads and shows the data in real-time, could be done but to my knowledge has not been done.
If I have been looking at the tool it is quite basic it does not show the progress data in real time it does not auto refresh the monitoring pages, in the reports I cannot filter and it is a problem if you have to keep track of multiple installations and backup groups . I think it does not mark a failure either when it waits for a report and it is not received. But it might work for me for now.
The truth is, seeing this software, I don’t know if it would be worth starting a new project that covers all the needs. I have been surprised by the simplicity of its development and how quickly you can start it without completely knowing how it works. In that sense a 10 for the developer.
Thank you very much for the tip.
Cheers,
Ramiro
------------------------------- Spanish
Si he estado viendo la herramienta es bastante básica no muestra los datos de progreso en tiempo real no auto refresca las paginas de monitorización, en los reportes no puedo filtrar y es un problema si tienes que hacer seguimiento de múltiples instalaciónes y grupos de copias de seguridad. Creo que tampoco marca fallo cuando espera un reporte y no es recibido. Pero podría servirme de momento.
La verdad viendo este software no se si merecería la pena comenzar un proyecto nuevo que cubra todas las necesidades. Me ha sorprendido la sencillez de su desarrollo y lo rápido que lo puedes poner en marcha desconociendo completamente su funcionamiento. En ese sentido un 10 para el desarrollador.
Muchas gracias por el apunte.
Saludos,
Ramiro
When it’s free, there are going to be limitations.
I forgot about this earlier but there is also the Duplicati Client which exposes a few more options. Maybe you can script & parse some of the reports/info you’re looking for?
Also, I presume you’ve already looked at the built-in reporting options but I’ll mention them in case you haven’t.
Providing you have somewhere to sent the reports to, you can then parse or look at the reports and get at least the last backup results but there is much more available.
If you have local access (VPN/RDP) to each server you can monitor the progress of the active job in the GUI (providing it’s a GUI based job). You can get the job status from the server component (what the GUI uses to show progess/status) but you will need to keep an open connection between yourself and each remote server along with create the script/program to query for data continuously and display it.
Keep in mind Duplicati is still in beta and it’s not quite ready for full on business type deployments. Better monitoring seems to be a frequently mentioned request that as far as I know, is not currently being worked on. I’ll add it to my ‘Duplicati: Things to look into.’ list but that list be growing by the day.
This is and I have it clear. I think Duplicati is a great app and I will be using the monitoring system to keep track of backups. I currently do it with my own application that processes the content of the emails and I would even dare to say that it is too complex for its purpose. What happens is that it processes emails that come from different backup tools. It is the only advantage.
If you mean to integrate with specific existing software, can you say something about that software?
If you mean to write new software, that might be good, because it might help you get Duplicati’s info.
How large-scale is this? Duplicati GUI only runs one backup at a time, unless this is CLI. GUI or CLI?
Scripting options can help run things, but what would you run? The main methods would either be CLI, monitoring the output, or polling over the GUI web interface. Latter is advanced but some have done it.
There are lots of monitoring solutions on the forum for specific integrations, but I think most monitor the backup result rather than progress within the backup. There can not be a percent completed even from GUI because only changed data is uploaded, and changes are only discovered as the backup operates.
dupReport takes this approach, but it doesn’t have progress because reporting email is at the end.
Does anything do progress? How? I don’t know all the tools, but email at end appears more typical.
I’m still trying to understand what you would like to receive or be able to look at to monitor progress.
Sounds pretty large scale. Duplicati definitely isn’t aimed at enterprise scale use right out of the box.
There’s also the reliability question, but it might do well enough to be a secondary business backup.
Reliability can be assisted (as can monitoring) with your own programs and scripts, but it’s an effort.
EDIT:
This and other variables give results directly to your script if you prefer that. No progress, just at end.
The software has been developed by us (php and javascript -MVC-) and alerts arrive from different backup programs (cobian backup, aomei, duplicati etc) from different computer infrastructures. This classifies them based on the content of the email body. Each review of the receiving email account and depending on the backup schedule is capable of detecting those that for one reason or another have not been executed, marking it as missing a report to classify as correct or erroneous. Since we want to migrate all backups to Duplicati, that is why I have been interested in the monitoring system.
Please note that we receive email alerts from different facilities running their backups independently, we intend to keep track of all of them. They come from different clients that have nothing to do with each other, so different backup sets can be running at the same time.
On clients running Duplicati at the moment you are doing it from the GUI. At the command level we consider it too complex and hard. For convenience we use the GUI in each of our clients.
I was surprised by the simplicity of the monitoring system, that is if many things could be added to maximize it. Nor would it be very necessary to have those backups that are running in real time (although it would be very interesting).
I didn’t know about that tool I’ll take a look at it.
By this means it is very complicated and on top of that with my bad English it may make things worse. Don’t worry, with the Duplicati monitoring tool that I discovered today, there is enough for now, although it could be greatly enhanced, with filters by groups, auto-refresh of screens, for the time being it would improve a lot. Then if you plan to make a backup and it is not as done, it will mark it as an error. These are ideas that I put at your disposal and that we already use in our monitoring system and that would be very interesting.
I have full confidence in the duplicati software and we have been using it since version 2.0.5.112 canary 2021-01-20 on clients and in the face of some problems we have restored without any problem. Like any other software, even if it is worth thousands of euros, it can also fail, right?
The monitoring issue that I have seen in https://www.duplicati-monitoring.com/ is more than enough, although it falls short since we can monitor many client installations. I simply wanted the possibility of enhancing it as soon as possible or developing a parallel software with the simplicity that I have seen in duplicati-monitoring.com with php, javascript and using MVC.
We have used the --send-http-url parameter locally with xampp successfully, but we were unable to upload the file to our remote servers.
You’re beyond me there, but for whatever it’s worth, the Duplicati GUI is in JavaScript + AngularJS, communicating to the Duplicati web server over its usual URL such as http:://localhost:8200
Duplicati Client is a Python script using the same undocumented-but-reverse-engineered API, and PowerShell script to run Duplicati GUI jobs by @JimboJones (who’s been replying) also uses API.
If you can screen-scrape for info, About → System info Server state properties has progress clues.
I haven’t heard of anyone trying that approach yet, but it exists, has less than the API, but is easier.
It’s hard to do right from scratch, but once setup in the GUI, an Export As Command-line is prebuilt.
False impression that CLI and GUI are inter-connected.has more history, and how those two relate. Different use cases and users may have different preferences, but CLI gives some progress words. Scripting languages typically have an ability to run things and study their status outputs in real-time. Mentioned originally because you wanted progress status, and text parsing is easier than API polls.
This is a third-party project, so is supported by its creator not the main forum, but does take requests.
Duplicati project has more than it can handle with the main work, thus it has reporting feeds to build on. Beyond that, I know of no plans to produce monitoring tools, but as mentioned, third parties have them.
Google search quickly saw several. We’ve already discussed two of the main ones, but there are more.
Duplicati-monitoring.com - central monitoring of multiple Duplicati instances + nice email reports
Announcing dupReport - A Duplicati Email Report Summary Generator (which can feed Apprise)
Apprise - Push Notifications that work with just about every platform!
Duplicati Dashboard - Monitoring solution
Get last backup status for PRTG sensor
Use Nagios for Duplicati Monitoring
Apologies to any I missed, and I didn’t even try to repeat the suggestions and links in the topics above.
That’s why very serious backup doesn’t rely on any one program, but I don’t know your reliability needs. Following the forum will show its rate of breakage, and https://usage-reporter.duplicati.com/ its use rate.
Canary is also not recommended for production because it gets new features, fixes, and possibly bugs. Occasionally it breaks and needs a respin, so consider watching release feedback from early adopters.
Good practices for well-maintained backups has other things you might consider to keep things healthy.
Duplicati-monitoring is great when it works. More often than not, I try to log in and the connection times out. I’m actually developing a similar service, but I’m just starting work on it now. I actually got here because I’m looking for documentation of the payload Duplicati it sends when you tell it to send your backup results somewhere.