How to improve my backup strategy? (Macrium+Duplicati+RAM-disk)

Hello everyone!
First I would like to say that Duplicati is absolutely awesome! I have yet to find anything that works closer to what I think how backups should work!

… Sadly that doesn’t mean I am 100% happy. ^^

First I’ll try to explain my situation and what I/we have done so far.
After that I have listed how I have set up Duplicati and a RAM-Disk for caching, so if you are just interested in that, skip ahead (to the “Duplicati”-Section).

I would be happy if anyone has suggestions on how to improve stuff for non-experts. :slight_smile:
EDIT: This post got quite long as I tried to explain where I come from. Maybe there is some value to it as a kind off “new user experience” or something like that.

===============
Pre-Duplicati

I don’t have a lot of experience in backing stuff up, so I’m sure there is plenty to criticize.
Also I want to preface that we can’t spend a lot of money on this. Let’s just say that 30€ for THE perfect program would be the limit. So I would like to keep it free-ish … small payments for really good stuff are fine (including maybe donations for Duplicati?)

My Girlfriend has a lot of personal data. Stuff for work, huge amounts of music, pictures and videos from when she was dancing professionally. On top of that large amounts of raw audio from her recording hobby.
All in all that is approx. 3+ TiB of stuff that I don’t want her to lose.

Until now our solution has been a kind of manual Raid 1? That is, using FreeFileSync occasionally to spread duplicates over all her internal HDDs. But that wasn’t very robust and pretty tedious.

I have an old QNAP-TS210 2-bay-NAS that she could use.
It is to slow to work from it directly but I think it makes for an okay backup destination.
But there is no way the backup runs through in a single sitting. And as I saw when I tried Macrium Reflect Free Edition for my personal computer: even differential backups can take preeeetty long and if the connection is a bit flaky, everything is back to zero.

So I searched for something that:

  • is free or cheap
  • can deal with slow connections
  • can quietly resume backups after interrupted connections or PC-restarts
  • can deal with large datasets
  • can do incremental backups to make regular backups in this setting feasible
  • can consolidate the backup chain because our budget and drive space is very limited and there is no space for backup rotations
  • is easy enough to use for her not to be totally scared about touching anything
  • restoring backups is easy enough so that she can restore files when I am not around (which can sometimes be months)

So my first try was Macrium Reflect Free Edition. But the differentials were bigger and slower than I liked and there is no consolidation.

My second choice fell on Aomei Backupper as it can do incrementals and is user friendly … but we realized very quickly that the program is very unstable … at least on her system. Backups can freez after a couple hundred GiB and sometimes it seemed to break the the explorer (including taskbar!) so much that a hard-reset was required. Also when Aomei encountered any problems we weren’t able to fully terminate it which meant that the system would hang mid-shutdown, requiring a hard-reset as well.

Then I tried Veeam Backup & Replication Community Edition and I thought that this program was an absolute overkill as it seems to be targeted at servers and backing up large numbers of VMs. Nevertheless setting up a backup was mostly straight forward except that I couldn’t get it to actually do the backup. From what I read in their forums it’s some problem on local machines when admin accounts try to backup themselves or other admin accounts? The solution seemed to be some registry modification regarding UAC or something like that. But my girlfriend said that I am not going to modify the registry to sneak around some account protection thingy and I accepted that, so Veeam had to go … which it didn’t fully because apparently a clean install of Veeam is incapable of uninstalling itself without errors.

Then I discovered HardLinkBackup and loved the concept! Unfortunately the free version is very limited. I might have been tempted to pay for it, but unfortunately it cannot resume backups. The initial run has to be in one sitting. Which is a shame because having continuously incremental, compressed backups and being able to restore files just by using the file explorer is a massive plus for usability for inexperienced users!

Backup Service Home 3 seems to be very similar but way more complete than the free HardLinkBackup. Unfortunately it also cannot resume backups. Probably worth checking out for everyone who needs very simple backups on smaller datasets!

================
Duplicati

Other than doing system-images (I’ll probably use Macrium for that) Duplicati seems to do everything I want!

With a few exceptions.

Her PC has a small OCZ-SSD as the system drive and I feel pretty uncomfortable writing multiple terabytes through its temp directory when there is more than enough RAM available for caching the volumes.

So this is how I set up everything for a test on my system:

  1. install Duplicati and change “Duplicati.GUI.TrayIcon.exe” to always start as admin

  2. install ImDisk

  3. copy “create_RAM_Disk.bat” and “destroy_RAM_Disk.bat” (see below) to the Duplicati installation directory (they require Duplicati to be started as admin)

  4. deactivate autostart for Duplicati in the task manager (because it can’t start Duplicati as admin?)

  5. set up task scheduler to launch Duplicati with elevated privileges on log-in

  6. open Duplicati web-interface and paste my “standard settings” (see below)

  7. set up backup plan including “–alternate-destination-marker” and “–alternate-target-paths” for more robust external HDD detection on my system.
    (By the way, these options seem to get deleted every time the backup-configuration-panel is opened in the web-interface. Each change requires re-applying them.)

===================================
create_RAM_Disk.bat

imdisk -a -t vm -s 4G -p “/fs:ntfs /q /y” -m Z:
label Z: RAM_Disk

===================================
destroy_RAM_Disk.bat

imdisk -D -m Z:

===================================
Standard Settings

–asynchronous-upload-folder=Z:\Temp
–asynchronous-concurrent-upload-limit=10
–blocksize=200KB
–use-block-cache=true
–concurrency-compressors=4
–dblock-size=100MB
–hardlink-policy=All
–snapshot-policy=Auto
–synchronous-upload=false
–thread-priority=belownormal
–usn-policy=Auto
–run-script-before-required=C:\Program Files\Duplicati 2\create_RAM_Disk.bat
–run-script-after=C:\Program Files\Duplicati 2\destroy_RAM_Disk.bat

=====================
Final Thoughts

Now, I think for non-tech people this is quite the setup process and I really wonder if there is a simpler way to achieve what I want?
Different steps, other programs?
(I am aware that there are many things that I could improve by spending money. Better SSD, so I don’t need to care about caching. More HDD-space for RAIDs and other backup paradigms. Better NAS. Proper, paid software … although that would have been a total fail if I had bought the full version of Aomei Backupper as it completely disintegrated itself after a couple days of use.)
(I am also aware that some of the problems I had may be down to specific issues with her system … but honestly … every Windows installation develops some kind of weird issues after some time. I expect any backup software to be robust enough to not explode for unknown reasons when the system works fine otherwise. I want my girlfriend to install a backup-program, not to re-install EVERYTHING (without even having proper backups).)

That said, I am super happy that I found Duplicati and I hope the project stays healthy and progresses quickly. :smiley:

Big THANK YOU to everyone who even read through half of all of this!
And sorry if my english sounds funny or if I wrote something stupid. I’m not native to the language nor to the topic. ^^

PS: Is “Support” the correct category for this post?

Welcome @A5tro thanks for the feedback.

You mention having trouble getting the “advanced options” to stick. This is a known issue. To work-around this, create the job without any advanced options save the job, then edit the job and add the advanced options in Step 5 and they will persist.

Duplicati will launch with as an Admin but if you want it to happen without the prompt you’ll need to lower your UAC settings to be less restrictive. In Win10, Settings > Control panel (search for it) > User Accounts > Change User Account Control Settings. By default the slider is at the 2nd setting from the top, set it to the 3rd or 4th setting to remove the UAC prompts.

You shouldn’t need to use the Task Scheduler, you’re just by-passing the UAC prompt.

2 Likes

Thank you JimboJones!
I didn’t realize that those markers were also available in step 5. I always added them in step 2. Now they stick!

Hm, I’ll test the normal autostart again then. I didn’t even get a prompt before …

So, I deactivated the planned task in the task scheduler and re-activated Duplicati in the autostart-tab in the task manager.
I rebooted … and nothing happend.
When I opened a browser that still had the web-UI tab open it wasn’t able to load it. So it looks like it completely fails to start.
This is the case on both of my machines. (Laptop and Desktop)

This seems to be the same / a similar problem? https://forum.duplicati.com/t/autostart-as-admin/10626?u=a5tro

Wow, what a can of worms that just opened up but at the end of it I’ve found a workaround or at least identified the issue.

Given I rarely reboot my main (Win10) workstation, I did and what to you know the Duplicati tray didn’t load, just as you explained. Hmmm… check where it’s supposed to load from which is shell:startup and there is the Duplicati shortcut which should work… Set to run as administrator, no change, try loading Duplicati from shell:common startup… Same thing.
Drop the UAC to the very bottom, reboot… WTF, still nothing. Add a string value in HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run that points to “Duplicati.GUI.TrayIcon.exe”, that’s gotta work, nope. Maybe HKLM\SOFTWARE\Microsoft\Windows\CurrentVerstion\Run … OMG, that didn’t work either. At this point, Task Managers Startup tab was showing 4 copies of Duplicati, all Enabled, not a single one of them running… at this point this makes no sense to me whatsoever.

So what’s different between “Duplicati.GUI.TrayIcon.exe” and every other “example.exe” program that I can get to load, no problem using any of the above methods… the two extra periods! Yes, the filename of the exe file. I renamed “Duplicati.GUI.TrayIcon.exe” to “DulicatiTrayIcon.exe”, reboot and holy cow the tray icon was running just like it should and the GUI page loaded fine.

This does not occur in Windows 7 btw but it does Windows 10. I’m also not sure if renaming the exe will cause other side effects but it seems to work.

Well that’s certainly weird!

So a shortcut to Duplicati.GUI.TrayIcon.exe works fine in startup. If you change the shortcut properties to run it as admin, then it no longer works. But if you then rename the exe and update the shortcut target accordingly, it starts working again? (and successfully runs as admin?)

It’s unclear to me if you still have the UAC slider set all the way down or not. (By the way the lowest position on that slider behaves differently in Win7 vs Win10.) Normally I don’t advise changing that slider just to try to get Duplicati to run elevated as the UAC setting affects everything system-wide.

Further details, UAC has been set back to it’s default value (2nd notch from top). The renamed exe is set to run as an administrator. Admin rights set by right-clicking on Desktop icon > Properties > Click “Open file location”, right-click on the exe > Properties > Compatibility > Click “Change settings for all users” > Click “Run this program as an administrator”. Reboot and Duplicati loads as it should. This is the same process I applied to the original file without success.

The “need” for elevation only stems from using VSS (–snapshot-policy=on) otherwise it did work fine non-elevated but once elevated this was the only way I could get it to load on startup.

As a network admin I have a real love hate relationship with the UAC, I often still disable the damn thing on principle alone but in this case adjustments to the UAC are unnecessary for the issue at hand.

I wonder if the “.GUI” is being seen as the extension when elevated for some reason. While not a common issue these days, extra period issues do still rear their ugly head every once in a while. When I get some time later, I’ll to replicate the issue with another app by giving it a something.xyz.somethingelse.exe file name and see if it will run elevated.

I’ve been trying to get notepad.exe to come up elevated, but tested with Advanced button and then check:

image

It would never come up, either in the all users or per-user Startup. Without option check, it comes up fine.
Tested on both an Administrator and a Standard User account to see whether the “dot” effect held for me.
Put UAC down as far as possible. Still no joy. I didn’t know your method of getting to administrator though.

Apologies to OP for getting off track. Personally I run Macrium Reflect Free to USB, Duplicati to the cloud.
Macrium doesn’t run often for me. As an image backup, I would sometimes run before Windows upgrade. Duplicati is file-oriented, and wouldn’t be quite what you want if a Windows upgrade trashed everything…

Typically it’s recommended to have at least two backups, with one remote in case local disaster happens.
Using different backup programs guards against program failures. Duplicati is pretty good, but not perfect.

Every program has some issues. Duplicati does have a pretty good feature set though, and price is good. Although it’s pushing your price limit to license the GUI, it’s possible Duplicacy can also do what you want. Other payware ones cost more, depending on how you calculate. Some free ones lack features, e.g. GUI.

We usually try to limit the backups to a few million blocks, otherwise database work can become too slow.
Maybe that should be 1 MB or more, however if things (including DB Recreate) are fast, maybe you’re OK.

That presumes the Desktop icon has been updated to point to the renamed copy of the .exe, if not start at “right-click on the exe” (the renamed exe) and follow down from there. I also tested with that checkbox in Advanced Properties but it made no change.

???

Not the issue here, because the .exe is notepad.exe. Shortcut %windir%\system32\notepad.exe
I’m trying to test like the “no problem” case below, except more extensively than in previous tries:

Specific method to set up such standard Windows programs was to find it in Explorer, right-click
Send to Desktop (create shortcut), then move that to the startup folder, e.g. shell:startup for user.

Care to see what you can do with notepad.exe with modified filename or the original like my test?

If a reliable, simple, and safe way to elevate Duplicati from shortcuts is found, that would be nice.
This could maybe even be put in the installer. There is already a wish for a nicer Service installer.
Elevated Tray Icon is a little nicer though. A Service needs a separate Tray Icon if one wants that.

Why not just try it renaming the Duplicati.GUI.TrayIcon.exe? It seems easier to test the specific case than to try an recreate the whole issue with something else but we can try it.

Head into %windir%\system32\ locate notepad.exe, copy it and paste it (you will need to click continue) the copy will be named notepad - Copy.exe, right-click on the copy and rename it to notepadTest.exe. From this point on “the file” means the copy of notepad.exe. Right-click on the file and go to properties, next on the Compatibility tab click Change settings for all users then add a checkmark beside Run as administrator, Ok your way out of the properties. Make a shortcut to the file and put it into shell:startup, reboot and see that notepadTest.exe runs on startup (it should).

Now that we have a test base remove the shortcut from shell:startup, rename the file to notepad.GUI.Test.exe, create a new shortcut and put it in shell:startup, reboot. If my theory stands, the file will no longer launch on startup.

I think I’ve covered things but I’m running late for a meeting and don’t have time to test it myself so I may have missed something, I’ll give it a test when I get back this afternoon.

For me, renaming the exe lost the “Run as administrator” on the .exe, so it came up unelevated.
Renaming it back to original found the checkmark still there. Maybe registry stores it by its path?
Regardless, I suggest using Task Manager’s Elevated column (or another check) to be certain.

After trying for a long time, I still haven’t found a recipe for getting Startup and also an elevation…
Tested more Windows programs, tested Duplicati TrayIconTest.exe and TrayIconTest.exe.config.
Windows 10 Version 21H1, Administrator account, shell:startup, shell common:startup, UAC off.

@ts678
Thank you for your suggestions! I read that larger blocksizes are recommended for larger backups but as I didn’t know what “large” means I went for twice the default initially, as I also read that increasing the blocksize also hurts deduplication performance? Now I increased it to 1MB though.

The renaming-to-run-as-administrator did not work immediately for me, so I went with the task-planner approach to get her backup going.

Some hours and 500+GiB into the backup I realized that it was getting corrupted (uncompleted volumes at destination) and after some reading on the forum I realized that repairing it would take just as long as starting fresh and that it wouldn’t fix the root cause.
From what I was able to understand it was most likely a problem with SMB and that switching to NFS was recommended, which is not supported in Win10 Home (I think?), only in Pro.
So I tried FTP which seems to run fine for now. The initial backup is still going, so I hope for the best this time.

I want to thank everyone for their time and dedication :blush:

1 Like

I know this thread is old but the orginal post settings contain a mistake and I just found this thread and it helped me set up the standard settings, and I guess that if it was useful for me, it will probably be for others too (inc how to get it right).

The problem is with the syntax. when you enter the above settings as text (in the GUI interface) you need to remove the dash in the beging, so instead of “–asynchronous-upload-folder=Z:\Temp” you write “asynchronous-upload-folder=Z:\Temp” - when you save it it will add the two required dashes in front of each entry

so the settings script should be:

asynchronous-upload-folder=Z:\Temp
asynchronous-concurrent-upload-limit=10
blocksize=200KB
use-block-cache=true
concurrency-compressors=4
dblock-size=100MB
hardlink-policy=All
snapshot-policy=Auto
synchronous-upload=false
thread-priority=belownormal
usn-policy=Auto
run-script-before-required=C:\Program Files\Duplicati 2\create_RAM_Disk.bat
run-script-after=C:\Program Files\Duplicati 2\destroy_RAM_Disk.bat