Using Duplicati for the entire System partition after a barebones drive image

Support
1

Imagine something like this wild strategy:

  • you install your OS on Drive0.
  • before installing any programs or configuring anything, make a small full system image of Drive0 with Clonezilla / Rescuezilla or just any vhdx.
  • forget about the system image, lock it away in safe place(s) and don’t make any new ones.
  • set up Duplicati for the “C:” partition (the OS partition of Drive0), for EVERYTHING in “C:”, and back up to some LAN location.

When a disaster happens:

  • restore the Clonezilla/Rescuezilla for Drive0 just to get the partition scheme & hidden partitions etc.
  • boot from some live CD, install Duplicati.
  • restore onto the Drive0’s “C:” partition, everything from the Duplicati backup, simply overwriting anything that might clash.

Why do this?

  • There’s no such thing as an encrypted & compressed VHDX / AVHDX, that you can back up incrementally via internet to a (untrusted) remote location and often, and do this while the host OS is running, from the host OS. Not in Open Source anyway; A..onis or P..agon or V..am or whatever can. But 5, 10, 50 years from now you’ll find out they are no more or worse; but Duplciati is, and vhdx is.
  • Nobody wants to boot into Clonezilla every other day and upload it offsite.
  • Duplicati can (shadow)copy everything so conveniently, and all you really need from a system image is the partitioning and the hidden / boot partition(s).

Ok, so anyone have any concerns with this? In Windows? In Linux? Because if this works, and it should, we solved a major disaster-recovery hassle, so don’t be eager to dismiss it.

(if / when you also run things like docker on the host you’re backing up from, that you may want to include, you can run a pre & post script in Duplicati to make sure they’re off during the backup.)

2

Hi @Doq, welcome to the forum :waving_hand:

There are some cases where this will not work. Some applications (I recall Internet Explorers cache files way back) will write specific sector locations into the files and expect the data to be there later. When you restore with Duplicati, it has no clue what the disk layout is and will happily restore files into the filesytem, unaware of any sector mappings.

It might “generally work” but I would not depend on it.

A harder way forward is to make sure you have backups of data and configurations, so you can ignore the disk and simply install an OS on a clean disk. Tools like Docker and Ansible can make this automated.

Downside is that you need to verify and test that you have everything included. Upside is that your backups are smaller and you can easily upgrade to latest version of the OS.

3

Thank you. Unfortunately

  1. can’t really treat your personal OS e.g. ‘as a dockerfile’ or ‘just not care & keep the OS stock’. It requires too many personal (ux, functionality, privacy) tweaks (that often can’t be done via commandline) that are too artisanal and morph across updates.
  2. it’s very impractical or impossible to split your personal files from your OS. (how do you strip all configs, app data, software assets & manual asset libraries, documents etc from the OS and reform them on a new OS?)

So it’s instead very interesting to hunt down any scenarios that might break my idea.

Duplicati

  • uses VSS (Volume Shadowcopy Service) on Windows, and Logical Volume Management (LVM) on Linux,
  • preserves file permissions.

The initial minimal drive image

  • saves all the partition structure and hidden partitions.
  • At least in Windows, the hidden partitions (EFI System Partition, Recovery Partition), remain static, UNLESS: you install additional OSes / dual boot, or you install a major feature windows update (which as a fan of windows LTSC, I don’t have or want feature updates).

So afaiks then the only issues are:

  1. Containers mounted at the time of backup: just temporarily dismount them, or back them up another way.

  2. Sector dependent data like the IE cache example, and any weird disk buffers? This doesn’t sound like it would be a problem any more; at least NTFS is a sector independent logical addressing system. But I’m no expert.

Of course we can also just try this strategy but also still maintain proper constant image backups just to be sure while “screwing around”.

4

If you’re willing to try it some, you might find some. If you’re not, it puts you at risk, however arguably the fallback is the usual of reinstall the OS then try to assemble everything else from a generous backup. I’m surprised this gets so few questions, because it’s actually pretty technical and potentially depends on the specific apps.

Image backups work well for this, backup everything, run fast, but IMO use lots of space. Even a hard drive dedup rearranged enough that an incremental got huge.

A hybrid image/file scheme is interesting, but seems like it needs a lot of analysis. NTFS has much data on, say, the C drive that might not even be visible to backup.

NTFS System Files is a start, and the left bar adds other things one can fret about. Possibly some don’t matter. Others might. Some, like the MFT, might recreate fine.

There are some specialized filesystem data that Duplicati doesn’t do well on I think. Sparse files, alternate data streams, and reparse points come to mind. This isn’t an issue limited to the hybrid approach (and actual impact is arguable), but I mention it because it’s probably lower fidelity than an image backup, if that’s your benchmark.

I “think” (not sure) Linux has less internal filesystem data, but it has specialty files, for example sockets and pipes, that don’t hold data-at-rest. I’m unsure of handling.

VSS does not automatically stop all open and locked file issues. It needs help from applications (and maybe Windows). Linux has nothing similar, but you could maybe live with a crash-consistent view. A clean stop is cleaner, but is more of a nuisance.

I’m not sure if it preserves user and SID. I think Linux Duplicati has user and UID. Windows image backup should avoid the issue of fresh install assigning new SID. After that, maybe the file restore will write newer copy of the registry on top of old.

On any sort of drive write, BitLocker may have to be dealt with, but there are ways.

I also looked at the boot path and thought that maybe it would work, but it can vary.

Duplicati itself can use Windows USN Journal, but pretty easily goes into a fallback (complete scan), and might just fall into that when image data looks way unfamiliar.

OS version major upgrades might change internals and call for new image backup, however those don’t happen often.

I would think apps with absolute sector references would have trouble with defrag, so I’m not sure that’s common. I couldn’t find anything describing claimed IE issue.

I cited one site to look at for internals to worry about, but there are other sites too. Duplicati, being a file-oriented backup, only deals with the levels that it can get to.

The plan might work, with maybe known limits that a normal backup plan also has. Or possibly issues will show up very obviously. If you try this, be sure to test it well.

You could also categorize data, e.g. keep a constant file backup of critical data, leaving a broader less frequent backup as a better-than-nothing safety net on it.

I do an infrequent just-about-everything file oriented backup to USB drive in two different backup programs, and might advise on some things to add to excludes.

1 Like
5

Thanks a lot for the meticulous context! I’ll study further.

Yeah I’ll probably do it 2 ways: a couple incremental VHDs that I do less often of everything, and then also this duplicati-whole-system-partition strategy.

It’s a between rock and hard place kind of issue. I didn’t arrive at this wacky idea lightly. Got pretty fed up with probably every other strategy under the sun…