Newbie questions

Hi,

I am new to Duplicati. Some questions:

  1. can it eMail a report?

  2. if so, can it handle smtp servers that use OAuth2 (gmail, yahoo, etc.)?

  3. can it run pre and post events (programs of my own authorship)?

  4. if so, can data from my scripts/programs be injected into the tape report?

  5. is there a post event for reboot and shutdown?

  6. can it be set to housekeep and clean out old archives after a certain amount of archives has been reached?

  7. Are the archives in a standard format that any file reader can accesses?

Many thanks,
-T

  1. Yes.
  2. I am not sure about oauth2, but I have seen other threads about people sending email reports to gmail.
  3. Yes you can run custom pre- or post- processes.
  4. Not sure what you mean. Tape report? You mean email report? I do not know if you can include output from pre- or post- processes in the email report.
  5. Not sure what this means exactly. You want Duplicati to do a reboot/shutdown after the backup is complete?
  6. Yes it has very flexible backup retention options.
  7. Backups use ZIP and AES256 standards but they are not readable by other products. You will need to use Duplicati to restore the data.
  1. I still call them tape reports, although no one uses tapes anymore. I am asking if I can include my data with Dupliati’s reports

  2. I want the option to reboot or shutdown after a backup

  3. If I can not read the archives with a standard file manager, then it is a deal killer. So you can disregard the about questions.

Thank you for the response!

  1. yup can turn on less secure apps in gMail, but the user gets inundated with prompts from Google to turn it back off, so eventually, they break down and do it and you lose your reports. Best to just support OAuth2

Yes, if you want your data backed up in native format, Duplicati is absolutely not for you.

However, with native format backups what you miss out on is the efficient storage of multiple versions thanks to Duplicati’s deduplication feature. For example on my work PC here I protect 37GB of data. I currently have over 120 backup versions yet it only takes 70GB on back end storage.

Personally I do both. I use native format backups with limited versions but then Duplicati for very long backup history.

I usually have 20 times the backup space I need on a local backup disk. And proprietary backups are a nightmare to restore.

It’s important to note the backup structure is not propritary. Everything is available.

In fact it’s possible to restore the data without Duplicati at all.

An example is provided here with 1000 lines of Python code: duplicati/Tools/Commandline/RestoreFromPython at master · duplicati/duplicati · GitHub
Which can be used if Duplicati isn’t able to restore for some reason.

I also vaguely remember someone posting about writing their own restore script in go? Don’t quote me on that, though.

The closest I recall to that was where somebody had their database from an interrupted first backup (no dlist), and spent an hour writing a rough Python script. Restore Backup with only dblock and dindex files.

Above is extreme but just shows that it’s possible. Usual bottom-up is Independent restore program then Duplicati.CommandLine.RecoveryTool.exe if somehow the GUI stops working. If download stops working then you download files another way, decrypt with standard or Duplicati tools, and let RecoveryTool finish. These tools seem to actually get used very infrequently (based on forum reports), but it’s best they exist.

Occasionally (especially since this is still a beta) problems do occur where best way out is a new backup. Backup wisely, don’t run Duplicati as an archiver (then delete your original), do keep multiple backups, etc.

Recovering by using the Duplicati Recovery tool
How the backup process works
How the restore process works
Developer documentation
Local database format
All details are not yet spelled out (maybe someday this will improve) but it’s far from secret. Open-source.

Restore speeds are lower than simple formats (direct copy being maybe the extreme example of simple). There have definitely been people who want something direct. Windows File History comes pretty close to direct copy (it puts date information in the filenames, to keep versions separate). Some other ideas are at:

Create incremental backups without .zip or any compression

If you decide a more complex format is worth it, my quick supplement to other questions is that gmail can probably be solved and other benefits gained by having a third-party monitor send email from its own site: Duplicati-monitoring.com - central monitoring of multiple Duplicati instances + nice email reports

Process email body with script reminds me of “include my data”, though I’m not clear on exactly what it is.

Your “reboot or shutdown” could possibly go in a –run-script-after but the timing of things would have to be checked to make sure everything else you need is done by then, including any reporting you needed done. Extreme control can be had if you avoid the GUI, and just use command line to drive the backup, boot, etc.

1 Like

Not if you use Duplicati to do the restore! :wink: But I get it - some people are looking for backups in native format. Best of luck on your quest!

I have been is disaster situations at customer sites before. If I can not read the data with a standard file manager, I am toast. And Duplicati (or Macrium Reflect or any other micky mouse backup) would be no where to be found. The format must be non proprietary. And the data must be the original data, not fragments of incremental backups.

I think what you guys are missing is that the machine will be in a non-functioning state and the data must be recovered. I have seem too may situations where the customer’s back up was completely useless. Recovery required having EXACTLY that version of the backup software – which could not be found – running on a functional machine. Good luck with that.

So, I am still looking. I may have to write this myself, but I really, really am not looking forward to that. I just don’t have the time.

And I really need OAuth2 supported

Actually no, we are not missing that. We know that we will have to install Duplicati on a different machine to restore data (if the main machine completely dies). I have tested it and found it’s easy enough to not be a concern. I do have my configuration data, encryption key, etc. carefully logged in a safe place.

But like I said above I get the desire to have a backup in native format. Nothing beats that for ease of restoration. I don’t think you have to write anything yourself though. If on Windows I’d probably use robocopy. If Linux I’d use rsync.

You are missing the point. You had access to “the” version of Duplicati that actually would recover the data. I find on customer sites that “the” version, is so out of date it can not be located. I have to take into account the things will not be maintained after I install them.

Maybe other vendors have tight version revlocks? AFAIK any Duplicati 2 can restore for any prior one.

Duplicati 1.3.4 in 2013 makes backups that Duplicati 2 could not restore, so theoretical point is heard, however “can not be located” is seemingly fixable with a download or a copy of the software you keep.

If you have a “bare metal” situation, where it’s not even known if the “metal” works, how do you start?

Requiring only tools that ship with the OS is easy (assuming enough can be found to rebuild the OS), however writing your own seems to get back to the (seemingly not insurmountable) need to get it in…

I think I do not have a clear picture of the whole use case (from dead system up) on where backup fits, however there seems to be a picture here that’s not an image restore, but OS rebuild and file restores.

If customer is in a hurry for data, though, idea of restore (maybe robocopy?) to good system may work.

Good luck finding or building what fits your use case. We can try further, but aren’t connecting well now.

Yep. Only 1.3 and 2.X are incompatible. But they’re also two different pieces of software that basically just share a name.

Any 2.X version will restore your data.

Okay, you guys are thinking an I.T. aware /computer literate person. My customer would not need me if they had that kind of skill.

  1. they don’t keep anything around I tell them to do from an extended time period. A few years pass and “what is this?” and they pitch it in the bin. You can not assume a customer will act rationally. I have yet to find one that does.

  2. in five to ten years when Duplicati is on 12.x and they are still running 2.x, despite all the good intentions of the developers, you can not trust that you will be able to find a version that will restore their stuff, let alone tell which version they were using. Sometimes developers just get sick and tired of carrying around old defunct baggage and they pull the plug.

The above is why I require a non-proprietary format that any file manager can read. And for goodness sake, NO FRAGMENT TREADS of incremental backups. I need the full, intact file.

Then use robocopy.

Any solution that offers deduplication natively will not store your files in original format, so if you’re hunting for backup programs you should automatically dismiss those. Also dismiss ones that do delta level versioning like Cloudberry - it will store files in native format on the first backup but subsequent file versions will only be the changed delta portion.

Schedule robocopy with some sort of rotating target location so you have a decent backup retention. Hope they will actually monitor it so that they know it’s working. Storage-level deduplication (like even Windows Server) will help you store more versions for an even longer retention yet keep all the files in native format.

Good luck.

I have been using Cobian, but it has not been support for so long now that it is starting to become an issue, especially with it not supporting OAuth2. I don’t believe Deplicati does either.