Just curious, I noticed the --changed-files option. Does that mean that Duplicati can work with macOS’ fsevents service/DB? macOS keeps track of changes on the file system and this is meant for applications like Duplicati (changed since, etc.).
As I understand it, if it can provide a list of created / modified / deleted files then yes it should work.
I suspect you use a --run-before script to fetch the list and place it in the parameter (or env variable, or maybe text file to be passed in) and Duplicati will limit it’s processing to those files.
Ah, no, that’s not how I would want to use it. I was wondering about native support in Duplicati to get that info directly from the OS. Because it is Duplicati that knows when it has last done a backup and a workaround using scripts would be very brittle. This is something that is (afaik) normally built-in to the software (Chronosync does it, CrashPlan does it, most backup programs can do it afaik as it greatly speeds up processing). I actually was assuming Duplicati was doing it until I noticed the flag.
Duplicati recently added what I suspect you’re after to the Windows Release: 2.0.4.5 (beta) 2018-11-28 as “USN support on Windows” using option –usn-policy. I don’t use macOS, but I suspect Windows got this feature first…
USN seems definitely to be something else. I’m talking about:
One of the most powerful features of file system events is their persistence across reboots. This means that your application can easily find out what happened since a particular time or a particular event in the distant past. By doing so, you can find out what files have been modified even when your application is not running. This can greatly simplify tasks such as backing up modified files, checking for changed dependencies in multi-file projects, and so on.
Sometimes what it takes to get a feature is someone who’s motivated and capable of adding it with some help. You can see the rejoicing about the offer of that feature in Real-time backup. There are many other demands with not enough volunteers to get everything desired (including reliability and performance) as fast as wished.
Implemented backup based on changes recorded in NTFS USN journal #3184 shows the contribution of USN, while back in the forum thread there was some talk of Linux and Mac. Basically, still seem to need a volunteer. Feel free to send someone to that thread if you know anyone. There’s even an offer of guidance on the code.
Actually, that sounds like the same thing. USN is implemented as a sequence number on NTFS such that you can “list all changes since last ID”, which is what I understand is the same as what FSEvents does.
Now that USN support is there, all infrastructure is ready, so it is just a matter of plugging in the calls to retrieve the list of modified files.
The --changed-files option is planned to be used with a monitoring service. The idea is that “some program” (i.e. the Duplicati server) is running all the time and monitors the source directories. This monitor process then records a list of changed files, and when a backup is executed it inputs this argument to avoid the lengthy scanning process.
However, if the OS supports USN (or FSEvents), there is no need to go through the trouble of using --changed-files and monitor, as the OS does that already.
So short answer is “No --changed-files will not use MacOS’s fsevents service as that parameter isn’t intended for that type of functionality.”
Long answer is “Duplicati’s recently added USN support implements similar functionality for Windows so likely just needs a little additional coding by somebody familiar MacOS to extend that functionality to fsevents.”
OK. I clearly misunderstood USN and --changed-files.
By the way: I’d be happy to set a bounty on adding support for macOS FSevents. How can I do that?
Or, in the meantime, is there someone who knows how to use FSevents to feed --changed-files?
FSEvents would; reduce the overhead of scanning dramatically.
For a comparison, Carbon Copy Cloner can back up only the changed files in my Home directory which contains about 2.5M files, 600GB, in about 1.5 minutes.
Duplicati backs up changed files in a test folder of around 360K files, 2.5GB, in 4.5 minutes