I’m currently on the lookout for a program to handle an off-site backup for an archive server we’re managing. Tricky part here is that the archive server spins its disks up and down when files are requested. In general file transfer times are fine, but file access times can be lenghty while the disks spin up. We have noticed our current solution ARQ Backup takes a long time each run to determine changes which ends up degrading the system’s performance.
Ideally, I’m looking for a backup application that doesn’t do a full scan of the source folder every run to determine changed files.
I believe for example URBackup does a single full scan, then updates its internal database when new files and folders are added to the source. I’ll definitely be checking that one out, but I’m curious if Duplicati could also suffice.
Does anyone know how Duplicati determines which files have been changed or added since the last backup?
Duplicati has 2 modes, the default one where the source directories are scanned, and an optional USN mode that is only working under Windows (it’s a Windows feature).
Duplicati has no control on the USN mode, it’s managed by Windows, so you can’t really expect that the USN mode will solve your problem since it’s designed to make scanning faster, not to eliminate it.
How does this relate to scanning the filesystem for changes? Doesn’t scan keep the drives spun up?
Is the archive server where Duplicati lives, or is Duplicati scanning remotely somehow (not advised)?
Duplicati just waking up to do anything is probably enough to access its drive. It also has a database.
The question is how it finds the changes. I see mentions in its forum that USN is involved, so it may be similar to Duplicati scheme – and subject to the same limits. If they do it continuously, that might help… Journal has a finite size, so heavy changes can overflow it. OTOH fsutil usn could increase journal size.
Overflow could be prevented by continuous watching, but Duplicati has many jobs that run at their time, meaning it’s not designed to watch in the background for something that one of the jobs would backup.
If a full scan is needed, for whatever USN deficiency, an Informational level message can be logged:
2023-09-22 16:51:33 -04 - [Information-Duplicati.Library.Main.Operation.BackupHandler-SkipUsnForVolume]: Performing full scan for volume "C:": USN journal entries were purged since last scan